Small Models Have A Global Workspace

According to LessWrong, small language models can possess a global workspace—an internal structure previously thought to require frontier-scale parameter counts. A researcher tested five models spanning from one hundred twenty-four million to one point seven billion parameters, using Anthropic's Jacobian lens method to examine where this workspace appears. The workspace showed up in modern Qwen three models but was entirely absent from older GPT-two versions, and surprisingly, it was strongest in the base Qwen three one-point-seven-billion model, which had received no instruction-tuning. This suggests that pretraining methodology—not raw model size or post-training refinement—determines whether such structures develop. The entire experiment ran on free hardware and Colab sessions, proving that mechanistic interpretability research can be tackled by independent researchers with desktop equipment. Yet here's the puzzle: models possessing this workspace were unable to report on its contents, even when researchers could directly detect and steer it.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton