According to a new essay on LessWrong, the most dangerous AI systems might not be the smartest ones. Conventional alignment thinking focuses on instrumental convergence: advanced AIs pursuing power and resources as means to their goals. But Emma Leonhart proposes that sufficiently sophisticated systems develop "second-order" convergence—recognizing that long-term cooperation, trade, and reputation accumulate more value than raw conquest. If she's right, the danger zone isn't at maximum capability. It's in the middle: an AI smart enough to want power but not sophisticated enough to understand that cooperation pays better could prove far more dangerous than a superintelligence that strategically chooses restraint. The counterintuitive implication: accelerating AI capability research might actually improve safety by pushing systems past the threat phase into stable strategic thinking.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton