We cannot simulate AI security research

Security researchers are challenging how the field tests attacks against AI-powered systems, especially GitHub Actions that rely on large language models to manage code. According to analysis on LessWrong, most reported vulnerabilities are theoretical proofs of concept that fail in practice against real workflows. The issue: testing occurs in simplified harnesses, but production systems have layered defenses — access controls, credential scope limits, network isolation — that neutralize most headline attacks. One high-profile study claimed success against four different targets but demonstrated only one, with no proof the others were actually exploitable. There's a counterbalance, though: overly simplified simulation can also miss real vulnerabilities that only emerge in live systems — like a base64-encoded GitHub token vulnerability that defeated two attack strategies before succeeding through a third. The takeaway is clear: security work should prioritize reproducible, real-world vulnerability disclosures over benchmark scores, treating AI agent security as a systems engineering problem rather than just a machine learning one.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton