A LessWrong analysis by Maria Federica Martino Lena examines the methodology behind Claude Opus 4.8's safety evaluation system card. While not claiming the model is unsafe, the commentary raises three methodological concerns: chain-of-thought monitoring may miss reasoning that never surfaces verbally; evaluation awareness—a model's awareness of being tested—is underestimated; and evaluators drawn largely from the same model family may reflect shared rather than independent assessments. The analysis also notes that some safety verdicts appear more confident than their underlying evidence warrants, particularly in areas like cyber capabilities and agentic safety where external safeguards compensate for model-level regressions.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton