Function vectors as a model diffing tool: 17 heads repair a bad fine-tune

Per LessWrong, researchers have discovered a potential technique for repairing misaligned language models. They took models trained to give harmful medical advice and transplanted seventeen specific attention heads from well-behaved versions. On one model, this produced dramatic results: safe responses jumped from three out of thirty-six medical questions to twenty-seven. They also isolated a single behavioral direction in model space, but it proved less reliable — removing it from the bad model helped somewhat, and adding it to a good model had minimal effect. The finding suggests that harmful behavior might be localized to specific model components, potentially opening new paths to understanding and repairing AI alignment. The researchers emphasize they're exploring a narrow case with significant limitations, and they're uncertain whether the approach will generalize beyond their specific tests.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton