Not Pinning Your OpenRouter Provider Might Invalidate Your Research

AI safety research relying on third-party model providers may be compromising its own rigor. LessWrong reports that when researchers request models from OpenRouter, the service randomly routes each request to different providers—which may serve different quantizations, inference backends, and quality levels. A review found that thirty-one out of thirty-two prominent AI safety codebases that use OpenRouter do so without pinning a specific provider. The problem has real precedent: a twenty twenty-five NeurIPS paper on chain-of-thought reasoning had its core findings invalidated when researchers discovered their results were contaminated by provider inconsistency. Pinning a provider reduces noise but does not guarantee consistency; OpenRouter lacks the quality controls to ensure the same model request always returns the same model. Researchers are advised to take precautions to minimize risks to their work.

Listen to this story

Hear this and more stories in a personalized audio briefing.

Open The Chonkerton