According to an essay on LessWrong, the work of ensuring AI systems develop safely may fundamentally defy measurement in ways other challenges don't. Drawing on poker as an analogy: even the strongest professional has only about a five-percent edge despite enormous variance. Shaping history through AI alignment is far harder—we live through this emergence once, the variance is worse, and many obstacles are political rather than technical. Short-term signals—benchmark improvements, capability demos—can't reliably guide us. The Vietnam War's Whiz Kids optimized metrics that moved in the right direction while the conflict was lost. The recommendation: stop chasing what looks most promising now, and instead build a diversified portfolio of approaches—including unconventional ones—while developing better long-term forecasting methods to understand where we actually stand.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton