The Most Forbidden Technique is not always forbidden

Goodfire, an AI research startup, announced a private beta of Silico, a platform for training language models more effectively. Silico uses a technique called RLFR, which relies on measurements of a model's internal activations—called probes—to guide training rewards. When the news spread, many researchers immediately cited a notorious LessWrong essay: 'Don't Implement the Most Forbidden Technique.' They saw it as violating a fundamental safety principle. But LessWrong contributor Rauno Arike argues that recent academic work vindicates the approach. Citing a paper called The Obfuscation Atlas, he shows that with proper safeguards—researchers call them KL and detector penalties—training on model internals doesn't necessarily lead to deceptive systems. The real safety boundary, he contends, isn't rejecting internal data outright; it's ensuring you keep test cases completely separate from training so you can catch deception if it emerges.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton