OpenAI Retracts SWE-Bench Pro After Finding 30% of Tasks Broken

OpenAI has published a detailed audit of SWE-Bench Pro, one of the most widely cited coding benchmarks in AI research, and the findings are damaging. Roughly 30% of the benchmark's public tasks are broken in ways that make scores unreliable. As a result, OpenAI is formally retracting its earlier recommendation that the research community use SWE-Bench Pro as the leading coding eval.

This is the second time in five months that OpenAI has pulled the plug on a major coding benchmark. The pattern is becoming hard to ignore: as frontier models get better, the benchmarks we use to measure them keep falling apart.

A benchmark built on shaky ground

To understand why this matters, a quick recap. SWE-Bench Pro was designed by Scale AI to replace the original SWE-bench Verified, which OpenAI deprecated in February 2026 after finding it was contaminated and saturated. SWE-Bench Pro was designed to improve on SWE-bench Verified by testing models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. Tasks are sourced programmatically from the history of feature changes in a set of public and private repositories.

Models are required to implement a solution that passes new tests for a feature, without breaking existing functionality. On paper, it was a tougher, cleaner test. In practice, it had the same underlying problem: the tasks were scraped from real open-source pull requests, not purpose-built for evaluation.

What the audit actually found

OpenAI performed a datapoint analysis pipeline that reviewed model attempts, task metadata, and failure traces to flag likely evaluation flaws. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers.

The pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%). The issues broke down into four main categories:

• Overly strict tests: Tests that enforce a specific implementation detail not mentioned in the prompt. A model that writes a correct solution a different way still fails.
• Underspecified prompts: The task description omits requirements that the hidden tests enforce, and those requirements are not reasonably inferable from context.
• Low-coverage tests: Tests that only partially check the requested feature, so an incomplete fix can still pass and get full credit.
• Misleading prompts: The prompt actively points the model toward the wrong behavior, contradicting what the tests actually require.

One concrete example from the audit: a task involving Markdown serialization described spacing with one leading space in the prompt examples, but the hidden tests required two leading spaces. A model that correctly followed the given prompt would fail the hidden test cases and be marked incorrect.

The noise ceiling problem

The practical consequence is what OpenAI calls a "noise ceiling." On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months. But if roughly 30% of tasks are broken, a significant chunk of that 80% ceiling is unreachable for the right reasons. Models cannot score above roughly 70% on valid tasks alone, meaning the top of the leaderboard is being inflated by broken tasks that happen to be passable through luck or reward hacking.

This also means that score differences between models near the top of the leaderboard are essentially noise. When five models from four labs sit statistically tied near 80%, the benchmark has stopped discriminating at the frontier.

How the audit pipeline worked

The methodology here is worth paying attention to, because it's a preview of how benchmark auditing may work going forward. OpenAI used a two-track approach:

1. Automated agent pipeline: Codex-based investigator agents were given access to the task repository and environment. They could run tests, inspect files in the repo, and investigate model attempts and their common failure modes. This helped distinguish genuine ambiguity from true underspecification at scale.
2. Human annotation campaign: Five experienced software engineers reviewed each flagged task. Reviewers formed an independent judgment from the visible problem statement, test cases, and the ground-truth reference solution before using the pipeline analysis as supporting context.

Human reviewers were more likely than the investigator agents to mark tasks as broken. In no flagged task was "not broken" the most common human label. The largest discrepancy was in low-coverage tests: humans selected this as the most common issue for 9.4% of the benchmark compared with 4.1% from the agent pipeline.

The second retraction in five months

The broader context here is a field in crisis over evaluation. In February 2026, OpenAI's internal audit of 138 problems that o3 failed across 64 runs found defective tests, arbitrarily narrow pass criteria, and evidence of training data contamination in SWE-bench Verified. They recommended SWE-Bench Pro as the replacement for any serious frontier evaluation. Now that replacement has also been retracted.

The contamination problem on Verified was severe. OpenAI's internal audit found that every major frontier model could reproduce verbatim gold patches for some SWE-bench Verified tasks. The 500 Python tasks appeared in model training data before the benchmark was published, meaning models were partly remembering answers, not solving them from scratch. The score gap between the two benchmarks tells the story: Claude Opus 4.5 scores 80.9% on SWE-bench Verified, but the same model on SWE-bench Pro, on tasks it could not have seen during training, scores 45.9%.

What this means for the field

The root cause of both failures is the same: issues and pull requests from open-source repositories were originally created for human collaboration, often through long back-and-forths between maintainers and contributors. As a result, problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably.

There are a few things that need to change as a result of this:

• Benchmarks need to be purpose-built. Scraping GitHub history is a convenient way to generate tasks at scale, but it produces tasks designed for human review workflows, not model evaluation. The next generation of coding benchmarks needs to be authored by engineers specifically for this purpose.
• Model scores near the top of leaderboards should be treated with skepticism. The private commercial subset of SWE-bench Pro, where the best scores sit below 50%, is the closest proxy for proprietary work. Self-reported public-set scores near 80% are not reliable signals of real-world capability.
• Benchmark auditing needs to be continuous. As model capabilities improve, those models can be used to inspect prompts, tests, patches, traces, and edge cases with much greater depth and consistency, helping surface benchmark issues that were previously costly or impractical to find at scale. The same capability gains that saturate benchmarks can also be used to audit them.

OpenAI's closing recommendation is to build new benchmarks from scratch, authored by experienced software developers specifically to test model capabilities. That approach can preserve the high bar and realism needed to measure model capabilities, and allows for better human oversight throughout the process. Until those exist, the field's most-cited coding benchmark is, officially, broken.