According to LessWrong contributor William Kiely, frontier AI companies should use a straightforward framework when deciding whether to build the next powerful AI system. He builds on Anthropic's model, which considers three scenarios: optimistic, where safety is readily achievable; pessimistic, where it's impossible; and intermediate, where it's difficult but doable. Kiely's key argument: most intermediate scenarios should actually be treated as near-pessimistic. Here's the reasoning. In an intermediate scenario, a company believes it can probably reduce catastrophic risk through focused safety research. That word—probably—matters enormously. If the probability that safety efforts will fail is too high, the company shouldn't proceed. Kiely proposes that frontier labs and independent regulators should rigorously estimate the likelihood that their safety measures will succeed before building. If catastrophic risk remains unacceptably high, development should halt. It's a principle grounded in humility: acknowledge what you don't know, quantify the danger, and let that assessment guide decisions about building advanced AI systems.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton