Value Generalisation 2: The Missing Hole in AIs’ abilities

The AI Alignment Forum hosts a theory about why large language models perpetually seem on the verge of artificial general intelligence yet never arrive: they lack 'strong generalisation,' a distinctly human ability to reason adaptively in novel situations—blending situational awareness, symbol grounding, and long-term planning. This absence explains, the argument goes, why expert users extract far more from LLMs than casual ones, why these systems make seemingly avoidable mistakes, and why they pass benchmarks without actually learning the underlying concepts those benchmarks test. But the deeper concern is what the forum calls 'value generalisation': applying human principles and preferences to situations no human has explicitly addressed. Unlike empirical knowledge, values lack a ground truth to test against, making this especially hard. Without developing explicit, robust value generalisation, the piece argues, advanced AI systems risk becoming either ineffectual or dangerously misaligned with human intentions.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton