Why an LLM cannot accumulate concepts

LessWrong is reporting on how language models process concepts. A researcher named Zenya argues that while LLMs can accurately understand any single statement about a concept, they don't accumulate understanding the way humans do. Think of tomography: a single image from one angle doesn't reveal structure — you need cross-sections from many angles. Humans gather multiple perspectives through dialogue and thinking, building them into a coherent grasp. LLMs, by contrast, output each perspective independently and never weave them back together. To test this, Zenya took multiple LLM outputs describing the same concept in different words — each internally consistent — and fed them back together. They contradicted each other, even though they originated from the same concept, because they lacked a shared coordinate system or context. The finding: LLMs have no mechanism to accumulate understanding across interactions, which may explain why vulnerabilities like jailbreaks emerge when meaning shifts between contexts without notice.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton