Datacurve's DeepSWE Shows GPT-5.6 Sol Beats Claude at 2.5x Lower Cost
Datacurve's DeepSWE leaderboard now shows GPT-5.6 Sol, Terra, and Luna results — Sol tops all 13 models at 73%, with striking cost-performance gaps across the family.
- Datacurve's DeepSWE leaderboard now includes all three GPT-5.6 models: Sol (73%), Terra (70%), and Luna (67%).
- GPT-5.6 Sol tops all 13 models at 73%, beating Claude Fable 5 (70%) at less than 40% of the per-task cost ($8.39 vs $21.63).
- Luna outscores Claude Opus 4.8 (59%) at $3.03/task vs $13.22 — roughly 5–7.5x more benchmark performance per dollar.
- DeepSWE uses 113 contamination-free tasks across 91 repos, with hand-written verifiers that have a 0.3% false-positive rate vs 8.5% on SWE-bench Pro.
- Qualitative analysis shows GPT implements multi-part prompts more completely; Claude frequently ships only one branch of parallel requirements.
- All models run on the same
mini-swe-agentharness; benchmark data and trajectories are open at GitHub.
Datacurve just posted the full DeepSWE leaderboard results for OpenAI's GPT-5.6 family. The leaderboard now covers 13 models, updated with Sol, Terra, and Luna results. GPT-5.6 Sol leads at 72.7% , but the more interesting story is what the numbers reveal about cost-efficiency across the three tiers, and how DeepSWE itself differs from the benchmarks you're already familiar with.
What DeepSWE actually measures
Most public coding benchmarks are starting to lose resolution at the frontier. Models cluster within a few percentage points of each other, and the confidence intervals overlap. DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no internet access , measuring an agent's ability to autonomously resolve real-world coding issues end to end.
The benchmark is built around four design principles that set it apart:
- Contamination-free tasks: every task is written from scratch, not adapted from existing commits or PRs. The reference solution never enters the public GitHub record, so no model has seen the answer during pretraining.
- Broad coverage: 113 tasks spanning 91 repositories across 5 languages , TypeScript, Go, Python, JavaScript, and Rust.
- Real complexity: prompts are roughly half the length of SWE-bench Pro's, yet the reference solutions average 668 lines of code across 7 files , compared to 120 lines and 5 files on SWE-bench Pro.
- Reliable verification: verifiers are hand-written to test observable behavior, not implementation details. DeepSWE's false-positive rate is 0.3% versus 8.5% on SWE-bench Pro; its false-negative rate is 1.1% versus 24%.