Why study alignment interventions on pre-RL checkpoints?

According to researchers on LessWrong, one of the strongest intervention points for AI safety might come before the reinforcement learning stage that fine-tunes model behavior. Here's the reasoning: when models encounter novel scenarios beyond their training, they revert to patterns learned earlier—during initial pretraining, midtraining with curated data, and supervised fine-tuning. These early training phases embed behavioral patterns that persist afterward. There's another key factor: reinforcement learning shapes what models output, but not how they think internally. Different underlying reasoning patterns can produce identical behavior, so long as it satisfies the reward signal. This means the alignment properties built into early training actually do the real work—making these foundational phases crucial intervention points for ensuring AI systems stay aligned in unpredictable real-world situations.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton