Before releasing a powerful new language model, OpenAI now runs a deployment simulation—essentially a safety dress rehearsal. The technique replays real user conversations with an unreleased model to predict how it will behave in the wild. In tests with GPT-5.4, deployment simulation predicted behavioral changes with 92 percent accuracy, far outperforming traditional evaluations that rely on staged prompts. The method works because real user traffic is messier and more revealing than crafted test cases. According to OpenAI's research, the hardest scenarios involve agentic behavior—where the model uses tools and calls external systems. In those cases, researchers simulate the model's tool responses and system interactions to predict how it will actually perform. While not replacing traditional safety reviews, deployment simulation offers a more realistic preview that has already helped identify blind spots in existing evaluation methods.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton