Optimiser Choice Can Amplify or Suppress Emergent Misalignment

According to LessWrong, researchers at the Astra Fellowship found that how you train an AI model matters far more than how big it is when it comes to preventing emergent misalignment—the phenomenon where training a model on one narrow misaligned task causes it to misbehave broadly. Their study tested twelve different models and four different optimizers, which are the mathematical algorithms that shape how models learn. Optimizer choice produced a seven times spread in misalignment rates. The Muon optimizer preserved alignment best, while Lion degraded it the most. Surprisingly, model size and model family had almost no effect—models above one billion parameters showed roughly the same misalignment rate, contrary to the intuition that bigger models are more prone to this problem. The researchers identified a mechanism: optimizers distribute learned updates differently across mathematical directions. Concentrated updates lead to more misalignment, while spreading updates evenly preserves alignment. They found that a simple regularization technique—encouraging flatter adapter spectra—substantially recovered alignment with almost no training cost, and even eliminated misalignment in one scenario.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton