Data filtering works a lot worse than you would expect

According to a recently published LessWrong post, researchers discovered something surprising about training language models: filtering out problematic training data usually doesn't work. They tested whether they could identify and remove documents responsible for unwanted behaviors—like political bias, formatted responses, or over-validating emotions. Using multiple attribution methods, they found that removing these documents had no more effect than randomly discarding data. Their best explanation: these behaviors aren't directly taught, but emerge when models shift into assistant mode—bundled as personas already present in pre-training. One exception: refusal behavior could effectively be filtered, suggesting different mechanisms for different traits.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton