Reading into VLM hallucinations using the Jacobian lens

According to LessWrong, researchers used Anthropic's Jacobian lens to peer inside a vision-language model's reasoning about images. They found something surprising: when asked 'Is there a lamp in this image?' with no lamp present, the model says yes all thirty-nine times. But reading the model's internal state reveals it actually knows the lamp isn't there—hallucinated objects show up with roughly one-tenth the signal strength of real objects it actually saw. Flip the question to a forced choice—'lamp or dog?'—and the model gets it right seventy-seven out of seventy-eight times. The researchers suggest VLM hallucination isn't a vision problem at all, but a decoding one: the model has the information, but the question format prevents it from using it.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton