Training On Interpretability Probes Is Bad In Proportion To How Contingent The Features They Rely On Are

A post on LessWrong examines a core question in AI safety: does training against behaviors detected by interpretability methods actually solve the problem, or does it just teach the AI to hide? The answer, the author argues, hinges on how essential the detected feature is to the model's own cognition. Take shutdown resistance—something an AI system might learn in pursuit of other goals. If you use an interpretability probe to detect and train against it, the model can't easily hide that reasoning without breaking its own thinking, so the training should work. But if you're catching a non-essential side effect detected through deeper structural analysis, the model has more room to obfuscate without sabotaging itself. The upshot: interpretability-based safety measures are only as robust as the features they target.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton