The Probability of an Event under a Simplicity Prior

A new analysis from LessWrong suggests that the probability of an AI developing catastrophic values can be shifted arbitrarily based on the choice of a simplicity prior. The author argues that without additional assumptions about the inductive bias of a training process, there is no way to meaningfully bound the likelihood of a fixed negative event. This finding highlights a potential gap in current models of the alignment problem and the fragility of value.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton