Cognition's FrontierCode Benchmark Humbles Every AI Coder With a 13.4% Score
Cognition's FrontierCode benchmark asks whether AI code is actually mergeable, and today's best models score just 13.4% on the hardest tasks
- FrontierCode is Cognition's new coding benchmark measuring whether AI code would actually be merged by real maintainers, not just pass tests.
- 20+ open-source maintainers from repos like Celery, Budibase, and Mattermost each spent 40+ hours building tasks and defining merge criteria.
- Best model Claude Opus 4.8 scores just 13.4% on the hardest Diamond task set; GPT-5.5 scores 6.3%, Gemini 3.1 Pro scores 4.7%.
- Three novel grading methods introduced: Reverse-Classical testing, Code Scope enforcement, and Adaptive Classical Grading via a tool called
mutagent. - FrontierCode produces 81% fewer misclassification errors than SWE-Bench Pro by combining unit tests, rubrics, and LLM-based verifiers.
- Tasks are not public (to prevent contamination) but Cognition is opening evaluation access to all model creators.
Every major coding benchmark today measures the same thing: does the code pass the tests? That question was useful when models were struggling to write functional patches at all. But as AI-generated code becomes the dominant path to production, passing tests is no longer the bar that matters. The real question is: would a senior maintainer actually merge this?
Cognition, the company behind the Devin coding agent, just released FrontierCode, a benchmark built around that harder question. The results are humbling for every model on the market.
The benchmark gap no one was measuring
The problem with existing benchmarks is well-documented. The Model Evaluation and Threat Research group (METR) published an analysis finding that many SWE-bench-passing pull requests would not survive human code review. A patch can make every failing test pass and still be wrong in ways that matter:
- It might hardcode a value that happens to satisfy the test condition, fix the symptom without touching the cause, introduce a subtle regression in a code path the tests don't cover, or be structured in a way that makes the next change significantly harder.
- Researchers at the Software Lab found that 7.2% to 8.4% of patches accepted by SWE-bench as correct were actually functionally incorrect when evaluated against the full developer test suite, translating to an absolute overestimation of 3.8 to 5.2 percentage points in reported resolution rates.
FrontierCode was built to close that gap. It measures how well models can truly meet the standards of high-quality production codebases, going beyond functional correctness to ask whether a model's output reflects the judgment of an experienced engineer.
Built by the people who actually merge PRs
The benchmark's core differentiator is who built it. More than 20 world-class open-source developers built realistic, diverse, and challenging coding tasks from the repos they maintain, spending more than 40 hours per task. These aren't synthetic problems -- they come from maintainers of 36 flagship repositories including Celery (29k stars), Budibase (28k stars), Uppy (30k stars), and Mattermost (37k stars).
The maintainers didn't just write the tasks. They defined what "mergeable" means in their specific codebase, then distilled that judgment into concrete evaluation criteria. As Mattermost core maintainer Claudio Costa put it: "FrontierCode's unique value comes from the human experience encoded in its evals: years of judgment about what makes code high-quality and worthy of merging."
The benchmark evaluates code across six axes:
- Behavioral correctness -- does the patch solve the stated problem?
- Regression safety -- does it break anything existing?
- Mechanical cleanliness -- does it pass lint, build, and style checks?
- Test correctness -- do the agent's own tests actually capture the desired behavior?
- Scope discipline -- does the patch touch only what it needs to?
- Code quality -- does it conform to codebase conventions, follow sound design patterns, and remain readable?
Three grading techniques you haven't seen before
The hardest part of building this benchmark wasn't writing the tasks -- it was grading them fairly. Cognition introduced three novel evaluation methods to handle the ambiguity of real-world code review:
Reverse-Classical testing checks that the tests an agent writes are actually meaningful. When agent-written tests are run against the original, broken codebase, they must fail. This gives an automated, deterministic check that the agent understood the problem well enough to write an effective test for it.
Code Scope enforces restraint. A good PR should modify only what it needs to, without touching unrelated files or introducing unnecessary refactors. The scope criterion combines file-level allow/deny rules, line count limits, and LLM-based checks that verify the locality of a change within a specific part of a file.
Adaptive Classical Grading solves the problem of open-ended tasks having multiple valid solutions. Static unit tests are too rigid -- a correct solution can fail because it uses a different function name or error message. Cognition built a tool called mutagent that uses an LLM to surgically patch the test environment to align with the agent's implementation details, allowing rigorous, deterministic tests to run on open-ended solutions.
What the scores actually reveal
FrontierCode has three nested task sets at increasing difficulty: Extended (150 tasks), Main (100 tasks), and Diamond (50 tasks, the hardest). The results are stark.
The best performing model, Claude Opus 4.8, achieves a score of only 13.4% on the Diamond task set. Other models score significantly lower: GPT-5.5 receives 6.3%, Gemini 3.1 Pro 4.7%, and others even less. On the full Extended set, Opus 4.8 reaches 51.8% -- still far from production-ready by any reasonable standard.
There's also a sharp divide between closed and open-source models. Kimi K2.6, the best-performing open-source model, achieves just 3.8% on Diamond, 16% on Main, and 37% on Extended.
One interesting wrinkle: GPT-5.5 consistently uses up to 4x fewer tokens than Opus 4.8, achieving a better cost-intelligence tradeoff despite its lower raw score on Diamond.
The failure modes are revealing. In one example task requiring models to refactor warning logs in a C++ codebase, Claude Opus 4.8 consistently produced code that was behaviorally correct but idiomatically wrong -- mixing LOG_WARNING() and std::cerr in the same multi-line message, baking in an assumption about stream identity that a future refactor could silently break. It passes tests. A senior engineer would not merge it.
Quality control that takes the grader seriously
Rubric-based grading introduces its own failure modes. Cognition built an extensive QC pipeline to address them. Each task goes through a five-stage process:
- Design -- the task creator audits each rubric item and documents its rationale
- Hack report -- the author tries to game the rubric with a deliberately lazy or incorrect solution, exposing weak criteria
- Calibration -- the author writes four distinct solutions targeting a range of scores from 0 to 100%
- Review -- a pod lead reviews the full eval candidate through multiple rounds before a Cognition researcher conducts a final review
- Re-review -- most tasks cycle through multiple iterations before passing
The result is that FrontierCode produces 81% fewer misclassification errors than other leading benchmarks, meaning FrontierCode scores are the most accurate ranking currently available.
What this means for how we evaluate models
The broader context here matters. AI benchmarks saturate while production failures grow. SWE-bench Verified, once the gold standard for coding agents, now sees Claude Opus 4.5 leading at 80.9% -- a number that suggests near-saturation. FrontierCode resets that ceiling entirely.
The benchmark also challenges a quiet assumption baked into how the industry trains and evaluates coding agents: that functional correctness is a sufficient proxy for code quality. METR's time-horizon analysis showed that models perform well on short tasks but degrade sharply on longer ones -- suggesting existing benchmarks overweight quick fixes and underweight the architectural reasoning that defines senior engineering work. FrontierCode is explicitly designed to test that harder regime.
While Cognition does not currently plan to release the tasks publicly to avoid contamination, they are opening up evaluation to all model creators, positioning FrontierCode as an ongoing signal rather than a static leaderboard. For teams building or evaluating coding agents, it's the first benchmark that asks the question that actually matters in production.