According to researcher gwern on LessWrong, there's a fundamental paradox in AI: neural networks memorize data stupidly while human brains generalize intelligently, despite both being neural networks. gwern proposes that humans achieve this through extreme overparameterization—using massive numbers of parameters but training on small, carefully filtered datasets at high learning rates. This forces brains to generalize rather than memorize. Current large language models do the opposite: they minimize variance by memorizing broadly. If this hypothesis is true, training trillion-parameter models briefly at high learning rates and testing them on adversarial tasks could produce networks that generalize like humans—with major benefits for robustness and AI safety.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton