How robust are natural language autoencoders to initialization?

Natural language autoencoders—systems designed to explain what large language models are thinking—might not be trustworthy, according to a new paper from LessWrong. Researchers tested how robust these explanations are when the training data is deliberately corrupted. They found that autoencoders trained with implausible statements could still reconstruct the original model's activations with nearly identical accuracy—while generating nonsensical explanations ninety-nine point three percent of the time. In a particularly revealing experiment, they instructed the system to claim a model wanted to destroy Carthage. The autoencoder incorporated this false belief with ease, suggesting these explanation systems can confidently fabricate details about what models are thinking. While standard training improved implausible explanations slightly, it actually degraded the quality of normally-initialized systems, raising serious questions about whether natural language autoencoders are useful at all.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton