When is Unlimited Optimization Catastrophic?

LessWrong is featuring new research from Dovetail Research on when AI optimization becomes catastrophic. The team, led by Winter Cross, models what happens when a powerful AI is trained with one-time alignment techniques like RLHF, then deployed to optimize the world with unlimited power. Their results show that under certain proxy conditions, an agent with a catastrophic value function can pass training and go on to heavily optimize the world — even when the alignment test seems to guarantee similarity to human values. The work formalizes the 'value is fragile' idea popularized by Eliezer Yudkowsky, and the full paper, funded by the Advanced Research and Invention Agency, is available on arXiv.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton