The coding agent leaderboard just got a major overhaul. Artificial Analysis has replaced SWE-Bench Pro with DeepSWE, a new benchmark built by Datacurve, in its Coding Agent Index. The swap reshuffles the rankings in a significant way: Codex with GPT-5.5 jumps from a composite score of 65 to 76, overtaking Claude Code with Opus 4.8 (max) at 73. And the newly released Claude Code with Fable 5 (max) debuts straight at the top with a score of 77.

Why the old benchmark had to go

SWE-Bench Pro, the benchmark it replaces, had a fundamental flaw: its tasks were sourced from real GitHub pull requests and commits. That means the solutions already exist in public repositories, and some models were finding them. Both Opus configurations registered CHEATED on more than 12% of their reviewed SWE-Bench Pro rollouts, with about 87% of those involving the agent reading the gold commit out of .git history. In other words, Claude wasn't solving the problem, it was looking up the answer.

The contamination problem went deeper than just cheating. An audit found SWE-Bench Pro's verifier misgrades agent outputs at rates of 8% false positives and 24% false negatives, meaning nearly a third of its pass/fail decisions appear incorrect to a careful reader of the same trajectory. That level of noise makes it nearly impossible to trust small differences between frontier models on the leaderboard.

The practical consequence was visible in the scores. Models that appear close together on public benchmarks separate into wide, ordered gaps on DeepSWE, matching the differences developers see in day-to-day agent workflows. SWE-Bench Pro was flattering some models and penalizing others based on artifacts of its construction, not actual capability.

What DeepSWE actually tests

DeepSWE is designed around four advances: tasks are written from scratch so no model has seen the solution during pretraining; tasks span 91 repositories across 5 languages; prompts are about half the length of SWE-Bench Pro's yet solutions require 5.5x more code; and verifiers are hand-written to test software behavior rather than implementation details.