Debate with Self-Play Best-of-N Optimization

According to a new post on LessWrong, AI researchers are testing a protocol called 'debate' to safely train large language models. The idea: two AI debaters argue opposing sides of a question, and a judge decides who won. The researchers used this method to study how optimization pressure affects model accuracy, testing it on coding and math tasks. They found that deeper debates—with more rounds of argument and rebuttal—are more robust against models gaming the judge rather than answering correctly. Interestingly, training both sides simultaneously didn't help more than just optimizing one side against a fixed critic. The work uses inference-time optimization to test these protocols before committing to expensive training runs, offering a cheaper way to experiment with debate as a training signal for alignment and oversight.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton