Whack-a-mole with a broken hammer: does a model internally track its automaton state?

LessWrong reports that researchers probed a one point five billion parameter model, Qwen2.5, to see if it internally tracks the state of a simple login automaton. By training a linear classifier on the model’s hidden activations, they could identify the correct DFA state about eighty eight point six percent of the time, well above the seventy six point nine percent achieved using only the transcript. When the order of events was scrambled, the probe still outperformed order‑blind methods, reaching eighty point eight percent versus a maximum of sixty point four percent, indicating the model follows the sequence of actions. However, the model struggled to count beyond two failed attempts, confusing the locked state with two failures, and split‑half testing showed many probe errors were due to the classifier guessing rather than faulty model beliefs. The authors recommend running quick split‑half checks to separate probe mistakes from genuine model misunderstandings.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton