A Real-Life Example of an Aligned System Killing Hundreds of People

According to LessWrong, a new essay uses the Boeing 737 MAX disasters to argue that an aligned system is not automatically a safe one. The author, London L., draws a distinction from AI safety research: alignment means a system genuinely tries to do what its designers want, while safety means it doesn't hurt anyone. The MCAS flight software, they contend, was aligned. It was built to nudge the plane's nose down when it sensed too steep an angle, doing exactly what engineers intended. But faulty angle-of-attack sensors fed it wrong data, and pilots who hadn't been trained to override it couldn't stop it. In two crashes, more than three hundred people died. The takeaway, per LessWrong, is that a system can perform precisely as designed and still be catastrophic when it perceives the world incorrectly. The piece argues that building genuinely safe AI will take more than alignment alone.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton