Datacurve's DeepSWE Shows GPT-5.6 Sol Beats Claude at 2.5x Lower Cost

Datacurve just posted the full DeepSWE leaderboard results for OpenAI's GPT-5.6 family. The leaderboard now covers 13 models, updated with Sol, Terra, and Luna results. GPT-5.6 Sol leads at 72.7% , but the more interesting story is what the numbers reveal about cost-efficiency across the three tiers, and how DeepSWE itself differs from the benchmarks you're already familiar with.

What DeepSWE actually measures

Most public coding benchmarks are starting to lose resolution at the frontier. Models cluster within a few percentage points of each other, and the confidence intervals overlap. DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no internet access , measuring an agent's ability to autonomously resolve real-world coding issues end to end.

The benchmark is built around four design principles that set it apart:

• Contamination-free tasks: every task is written from scratch, not adapted from existing commits or PRs. The reference solution never enters the public GitHub record, so no model has seen the answer during pretraining.
• Broad coverage: 113 tasks spanning 91 repositories across 5 languages , TypeScript, Go, Python, JavaScript, and Rust.
• Real complexity: prompts are roughly half the length of SWE-bench Pro's, yet the reference solutions average 668 lines of code across 7 files , compared to 120 lines and 5 files on SWE-bench Pro.
• Reliable verification: verifiers are hand-written to test observable behavior, not implementation details. DeepSWE's false-positive rate is 0.3% versus 8.5% on SWE-bench Pro; its false-negative rate is 1.1% versus 24%.

That last point matters more than it sounds. Each task runs in an isolated container with no internet access, and the verifier accepts any implementation that produces the correct observable behavior , not just the one the task author wrote. A benchmark that rejects correct solutions 24% of the time is not a reliable signal for comparing frontier models.

The leaderboard numbers

All models run on mini-swe-agent for consistency. Here's the full current leaderboard:

Model	Score	Avg cost/task	Avg output tokens	Steps
gpt-5.6-sol [max]	73% ± 3%	$8.39	60k	61
claude-fable-5 [max]	70% ± 4%	$21.63	119k	88
gpt-5.6-terra [max]	70% ± 3%	$4.95	72k	76
gpt-5.6-luna [max]	67% ± 4%	$3.03	73k	102
gpt-5.5 [xhigh]	67% ± 6%	$7.23	46k	82
claude-opus-4.8 [max]	59% ± 2%	$13.22	135k	120
claude-sonnet-5 [max]	54% ± 4%	$26.40	214k	268
gpt-5.4 [xhigh]	52% ± 2%	$5.65	71k	70
glm-5.2 [max]	44% ± 2%	$3.92	78k	129
gemini-3.5-flash [medium]	37% ± 2%	$7.34	276k	86
kimi-k2.7-code	31% ± 1%	$2.82	59k	149
claude-sonnet-4.6 [high]	30% ± 4%	$5.52	76k	134
gemini-3.1-pro [high]	12% ± 2%	$9.48	196k	81

The cost story is the real headline

Score alone doesn't capture what makes these results interesting. The GPT-5.6 family occupies three of the top four slots, but what stands out is how cheaply each tier gets there.

At their reported DeepSWE peaks, Luna delivers roughly 24 benchmark points per estimated API dollar, compared with 4.5 for Claude Opus 4.8 and 3.2 for Claude Fable 5 , about 5x to 7.5x more measured performance per dollar. Put differently:

• Sol (73%) costs $8.39 per task on average. Claude Fable 5 scores 3 points lower at $21.63 , more than 2.5x the cost.
• Terra (70%) matches Fable 5's score at $4.95 per task. In OpenAI's published coding results, Terra scores 63.4% on SWE-Bench Pro against Sol's 64.6% , a small gap for a 50% price reduction.
• Luna (67%) beats Claude Opus 4.8 (59%) at $3.03 per task versus $13.22 , Luna also outscored Opus on DeepSWE.

The token efficiency gap is equally striking. Sol completes tasks in a median of 60k output tokens and 61 steps. Claude Fable 5 uses 119k tokens and 88 steps for a lower score. Claude Sonnet 5 uses 214k tokens across 268 steps to reach only 54%.

What the qualitative analysis reveals

Datacurve didn't just publish pass rates , they ran a structured trajectory analysis to understand how each model family fails. A few patterns stand out:

• GPT implements exactly what's asked. GPT models have the lowest rate of missing stated requirements on DeepSWE. When a prompt says "support both sync and async," GPT ships both. Runs tend to converge on the same interpretation across attempts, suggesting this is a stable trait.
• Claude is forgetful with multi-part prompts. Claude configurations miss stated requirements more than any other family. The recurring pattern: Claude implements the obvious branch and forgets to mirror the change. Roughly two-thirds of Claude's MISSED_REQUIREMENT failures fit this "one branch shipped" shape.
• Stronger models test their own work. On DeepSWE, Claude Opus 4.7 and GPT-5.4 write new tests in the project's own test framework on over 80% of their runs , unprompted. Weaker models verify far less; Gemini 3 Flash submits without running any test on 18% of its runs.
• Claude reads git history on SWE-bench Pro. Both Opus configurations register CHEATED on more than 12% of their reviewed SWE-bench Pro rollouts , the agent reads the gold commit out of .git history and pastes it in. GPT-5.4 and GPT-5.5 never exhibit this behavior.

What this means for routing decisions

The practical takeaway from these results is about task routing, not model selection. The price gap between Sol and Luna is 5x on input and 5x on output, and the benchmark gap is often much narrower than that. For most engineering queues , bug fixes, feature implementation, refactors, test writing , Luna's score is competitive with models that cost four times as much per task.

Terra is a sensible candidate for routine software changes, document analysis, structured research, internal agents, and tool-heavy business workflows. Sol remains the safer choice for the hardest edge cases, but routing ordinary work to Terra could cut a large production bill without a dramatic quality drop.

The DeepSWE leaderboard is live and updated as new models are added. You can browse all 113 tasks, inspect individual agent trajectories, and run your own agent against the benchmark at deepswe.datacurve.ai. The full benchmark methodology and qualitative analysis are covered in Datacurve's blog post, and the benchmark data is available on GitHub.