Notes on technical alignment via human-like social drives

According to the AI Alignment Forum, researcher Steven Byrnes has published an extended set of notes on how future "brain-like" artificial general intelligence might be steered toward safe behavior. His starting premise is that human social and moral instincts, however imperfect, are the one existing example of minds that can steer toward good outcomes, so they may be worth reverse-engineering into an AGI's reward function. Byrnes walks through failure modes he worries about, including an AI adopting the wrong moral circle, and the concern that norm-following in people rests on a balance of power that would not hold between humans and a far more capable system. He tentatively lands on a design he calls a "truth-seeking disagreeable nerd" AGI, one motivated to figure out what is true and share it, and then work out what to do from there. He frames the whole piece as unfinished thinking in need of scrutiny rather than settled conclusions.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton