OpenAI has published a detailed audit of SWE-Bench Pro, one of the most widely cited coding benchmarks in AI research, and the findings are damaging. Roughly 30% of the benchmark's public tasks are broken in ways that make scores unreliable. As a result, OpenAI is formally retracting its earlier recommendation that the research community use SWE-Bench Pro as the leading coding eval.
This is the second time in five months that OpenAI has pulled the plug on a major coding benchmark. The pattern is becoming hard to ignore: as frontier models get better, the benchmarks we use to measure them keep falling apart.
A benchmark built on shaky ground
To understand why this matters, a quick recap. SWE-Bench Pro was designed by Scale AI to replace the original SWE-bench Verified, which OpenAI deprecated in February 2026 after finding it was contaminated and saturated. SWE-Bench Pro was designed to improve on SWE-bench Verified by testing models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. Tasks are sourced programmatically from the history of feature changes in a set of public and private repositories.
Models are required to implement a solution that passes new tests for a feature, without breaking existing functionality. On paper, it was a tougher, cleaner test. In practice, it had the same underlying problem: the tasks were scraped from real open-source pull requests, not purpose-built for evaluation.
What the audit actually found
OpenAI performed a datapoint analysis pipeline that reviewed model attempts, task metadata, and failure traces to flag likely evaluation flaws. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers.
The pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%). The issues broke down into four main categories:
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves