Training AI to be better at correctness than persuasion

According to LessWrong, AI researcher Michael Dickens argues that we should worry not just about misaligned AI, but about something subtler: super-persuasive AI that's simply wrong. Even if an advanced AI is trying to help us, if it becomes extraordinarily good at convincing humans of incorrect arguments—especially on difficult philosophical questions—it could lead humanity down the wrong path. Dickens proposes a training approach using reinforcement learning on provable domains like mathematics. The idea: train AI to generate proofs, have mathematicians judge them, but crucially, penalize the model when mathematicians wrongly approve incorrect proofs. This trains correctness over persuasiveness. But the proposal comes with major caveats. It's unclear whether skill on provable math problems would transfer to fuzzy ethical questions. There's also a risk that teaching an AI what humans find persuasive—only to punish it for using that knowledge—could backfire and make it even more super-persuasive as a side effect. Dickens frames this as contingency planning, noting he still believes frontier AI development should be paused. But the underlying question remains urgent: how do we build AI systems that figure out what we should do without persuading us to do the wrong thing?

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton