Are AI labs pelicanmaxxing?

Researcher Dylan Castillo decided to seriously test whether AI labs had secretly optimized their models to draw pelicans riding bicycles—a joke benchmark Simon Willison had posed. Per Simon Willison's Weblog, Castillo ran forty-eight prompts across seven models three times each, using AI tools to evaluate the results. His finding: no evidence of pelimaxxing. The labs show no meaningful advantage at drawing pelicans, bicycles, or the combination—suggesting AI companies apparently have bigger things to optimize for than a silly benchmark.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton