P(doom) is a Dumb Meme

According to LessWrong contributor Max Harms, the AI risk community's fixation on 'P of doom'—the probability of catastrophic AI outcomes—suffers from a basic flaw: nobody agrees on what 'doom' actually means. Is it human extinction? A war killing ninety-nine point nine percent of people? An AI takeover that enables authoritarian rule? Economic displacement? The post lists seven wildly different scenarios, each defensibly called doom—from ten percent human death to gradual replacement by machines. Then there's timeframe ambiguity: five to ten years? This century? Heat death of the universe? Harms also surfaces a deeper problem. When experts adjust their estimates based on what others are saying—updating on perceived consensus—errors compound instead of canceling out. This creates false confidence in numbers that started as guesses. The core insight: a precise-sounding probability like twenty percent is meaningless without first agreeing on what we're actually measuring.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton