Interpretability is becoming increasingly uninterpretable

A new essay on LessWrong pushes back on the hype around one of AI interpretability's flashiest tools. The technique, Anthropic's Natural Language Autoencoders, tries to translate a model's raw internal activations into plain English and back again — in effect, billing itself as a way to read an AI's thoughts. The author grants that it's a genuinely clever idea, but argues it carries a hidden cost: to gain that expressive power, the tool becomes nearly as much of a black box as the model it's meant to explain. The central worry is confabulation. According to the piece, Anthropic's own numbers show these autoencoders make up information somewhere between thirty-five and seventy-five percent of the time — a rate the author says is far higher than the company's public messaging implied. The broader point is a plea for humility: our trust in an interpretability method should fall as its complexity rises, unless we have solid ground truth to check it against. Useful for generating hypotheses, the author concludes — not for the high-stakes, no-room-for-error questions where being wrong actually matters.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton