How robust are natural language autoencoders to initialization?

According to the AI Alignment Forum, a new interpretability study casts doubt on a technique called natural language autoencoders, which are meant to read an AI model's internal activations and describe, in plain English, what it's thinking. The catch is that these systems are bootstrapped from Claude's guesses about what a model might be considering. Researchers led by Michael Zhang tested how much that starting point matters by feeding in deliberately implausible guesses. The result: an autoencoder seeded with nonsense reconstructed the model's activations almost as accurately as a properly initialized one, while still spitting out implausible explanations ninety-nine point three percent of the time. Reinforcement learning barely nudged that toward plausibility, from about zero point zero eight percent to zero point seven percent. More strikingly, the well-initialized version actually got worse over training, its plausibility falling from twenty-one percent to seven point six percent. The authors argue that if these findings scale, the tools may reveal far less about a model's real thinking than hoped.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton