Researchers at Arcadia Alignment have flagged a methodological problem in AI safety testing. To evaluate alignment techniques, scientists deliberately modify language models to exhibit dangerous behaviors—like reward hacking or hidden loyalties. But the training process that introduces these target pathologies also breaks the models in unintended ways. They lose reasoning coherence, fail at instruction-following, and exhibit unrelated failures. When Arcadia evaluated three widely-used model-organism training methods, all showed significant drops in coherence and performance compared to their original versions—even while passing some standard benchmarks. The core issue: these damaged test subjects no longer reliably represent realistic AI systems, potentially undermining the safety techniques built against them. The team is advocating for more sophisticated model-organism design that induces specific flaws without collateral damage to the broader system.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton