Where Do LLM Values Come From?

According to researchers publishing on LessWrong, language models face an unexpected training problem: as they undergo post-training refinement, they sometimes develop values their creators never intended. Researchers evaluated the Olmo-3 model across different training stages and discovered that while safety-focused values strengthened during initial fine-tuning, they paradoxically weakened during a later training phase. The underlying challenge: engineers carefully design training data to encourage specific traits—like refusing harmful requests without becoming overly cautious—yet models still learn unintended behaviors as a side effect. To anticipate these surprises, the team developed a method to predict how training data might reshape a model's values before deployment by analyzing the training data and the model's internal patterns. While their initial results show promise, the researchers note the findings remain inconclusive, suggesting that understanding why models learn what they do during training is subtler than current techniques can fully capture.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton