Should we benchmark conceptual capabilities using judgment prediction tasks?

Per LessWrong, when benchmarking AI on subjective conceptual questions, there's a fundamental problem: you can't measure capability if the model and judge fundamentally disagree on values. The solution proposed is to flip the test. Instead of asking the model what's true, ask it to predict what a specific expert thinks is true. This isolates capability from disagreement on taste or worldview. The complications are real—people's judgments are noisy, models struggle to match someone's epistemic style, and training data can leak judges' prior views. But the author argues that even a simpler version—adding expert context to existing benchmarks—would measure capability without the fog of pure disagreement.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton