According to Google DeepMind researchers, model diffing agents are auditing tools designed to find behavioral differences between language models. Rather than testing against static benchmarks, these agents craft their own test cases to actively hunt for subtle, unexpected variations. The approach combines language model auditing with comparative model diffing. The team found their agents reliably surfaced differences that standard evaluations might miss—especially for rare or subtle behavioral changes. They validated their method using test cases with known differences. The work addresses a fundamental limitation in AI evaluation: conventional methods only find what you're looking for. DeepMind's tools aim to surface unknown unknowns in model behavior—the surprising things researchers aren't specifically testing for.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton