Data filtering works a lot worse than you would expect

According to the AI Alignment Forum, a natural safety idea just failed a stress test: if a language model picks up an undesirable habit during fine-tuning, can you simply find the offending training documents and delete them? Researchers Dohun Lee and J Rosser, working under Neel Nanda, tried exactly that on a scaled-down version of the OLMo-3 model. They targeted behaviors like heavy use of bold formatting, both-sides framing, a liberal lean, and reflexively telling users their feelings are valid. Then they threw the full toolkit at finding the responsible data — LLM autoraters, probes, activation methods, and gradient-based attribution. The surprise: filtering the top-scoring documents worked no better than deleting random ones. Even removing ten percent of the data barely dented the feelings-validation habit. Their explanation is that these traits aren't taught by a handful of examples but bundled into an assistant-like persona the model already carries — fine-tuning merely elicits it. The one exception that could actually be filtered out was refusal behavior.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton