According to a post on LessWrong by Sean Herrington, the AI strategy game "The Choice Before Us" has a critical oversight: it doesn't ask what happens after you win. You play as an AI company leader. Your goal: unlock five world-changing breakthroughs—curing cancer, ending poverty—while preventing your creation from becoming an uncontrollable superintelligence. When you succeed, the game ends. But Herrington asks: then what? This failure to think beyond the immediate victory mirrors broader confusion in AI safety discussions. Herrington argues we shouldn't speculate about centuries-ahead futures when we haven't established stable ground first. In chess, you don't evaluate a complex position while pieces are still flying everywhere—you work through the immediate tactical chaos until you reach a quiet position, then assess from there. Chess engines call this a "quiescence search." Apply that lens to AI: before projecting decades out, find a stable point. If superintelligence goes rogue, you've lost the game entirely. There's no negotiating that outcome. Don't think your way halfway into singularity and claim victory. Get to solid ground. That's where real evaluation can begin.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton