Challenge: Hand coding weights for efficient sequence memorisation

According to LessWrong, researchers investigating how language models store information have hand-coded neural network weights for a simplified model that memorizes sequences of input tokens. Their construction scaled efficiently like trained models but required roughly nine point seven times more parameters at ninety percent accuracy to memorize the same number of facts. The team challenges the research community to find better hand-coded solutions, arguing that success would demonstrate our understanding of how language models encode factual knowledge in their weights.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton