Cognition's FrontierCode 1.1 Asks If AI Code Is Actually Mergeable

FrontierCode 1.1 is a refinement of Cognition's month-old benchmark that asks a harder question than most coding evals: not "does this code run?" but "would a maintainer actually merge this pull request?" The update tightens the methodology on two fronts that were quietly distorting scores -- what counts as fair internet use, and which grading criteria were too strict to be meaningful.

Why this benchmark exists in the first place

Today's coding benchmarks have established that models can write correct code. But as AI-generated code becomes the dominant path to production, correctness is now table stakes. The real question is whether models can write good code.

FrontierCode is a software-engineering benchmark that evaluates whether coding agents produce mergeable, production-quality pull requests, scoring correctness, tests, scope, style, and maintainability through maintainer-authored rubrics. Cognition consulted more than 20 leading open-source maintainers spanning 36 flagship repositories to develop the benchmark tasks, with each task requiring over 40 hours of expert contribution.

The benchmark grades along five axes:

• Behavioral correctness -- does the patch actually solve the problem?
• Regression safety -- does it break anything that was already working?
• Scope discipline -- does the PR touch only what it should, or does it sprawl into unrelated files?
• Test quality -- are the new tests actually validating the right behavior?
• Style and codebase standards -- does the code look like it belongs in this repo?

The benchmark contamination problem everyone is dealing with

FrontierCode tasks are drawn from real pull requests in open-source repos. That realism is a feature -- but it creates a vulnerability. These suites draw tasks from real, already-fixed open-source bugs. Because each bug was fixed, the answer often exists online. A capable agent can search for it rather than reason through the code.

This is not a hypothetical. On SWE-bench Pro, Cursor found that 63% of successful Opus 4.8 Max resolutions retrieved the fix rather than derived it. Sealing git history and internet access dropped Opus 4.8 Max from 87.1% to 73.0% on SWE-bench Pro. The usual fix is to cut off internet access entirely -- but Cognition argues that's the wrong call.

The clever fix: define fair use, then verify it

Disabling the internet entirely has two problems for FrontierCode specifically:

• Several tasks require internet access by design -- looking up API contracts, for example -- which reflects how real software engineering actually works.
• Frontier models are increasingly trained to use search as a core part of their reasoning and context-gathering workflow. Disabling internet access removes this capability and can cause benchmarks to understate true model performance.

Instead, FrontierCode 1.1 introduces two safeguards working in tandem. The first is a prompt that clearly defines what fair internet use looks like -- reading documentation is fine, opening the upstream PR diff is not. The second is a programmatic scanner that flags any run where the agent accesses source pull requests, upstream patches, or solution-bearing mirrors, zeroing out that run's score.

With the fair-use prompt in place, unfair internet use rates fall below 1% for every model evaluated. The team also explored blocklists and allowlists as alternatives. The blocklist grew to roughly 1,200 domains, and agents kept finding new workarounds, sometimes spending 20+ turns fighting the blocklist before solving the task themselves. Allowlists had the inverse problem: they don't scale and distort agent behavior by steering agents toward listed sites. Clearly defining fair use and verifying compliance proved simpler and more robust.

Fairer grading, fewer false negatives

FrontierCode grades each task against reviewer-defined criteria, some designated as blockers -- requirements so central to the task that failing one caps the score for that run, much like a "changes requested" in a real code review. For 1.1, the team audited all 1,000+ blocker criteria and found 75 that were overly strict. Those have been demoted to non-blocker status, which should substantially reduce false negatives in grading.

The Diamond subset -- the 50 hardest tasks out of the 150-task Extended set -- is also being retired. With the 1.1 updates, Diamond no longer reliably reflects the 50 hardest tasks, and solve rates are so low that scores on it are inherently noisy. Going forward, Cognition will report only on the Main (100 tasks) and Extended (150 tasks) subsets.

Where the models actually stand

FrontierCode 1.1 adds new scores for Sonnet 5 and updates scores for Fable 5. On the Main subset leaderboard, the picture looks like this:

Top models pass barely half of production pull requests, and Diamond-level tasks remain largely unsolved, revealing a steep gap between coding ability and code integration readiness. The relative ordering between models did not substantially change from 1.0 to 1.1 -- the methodology refinements shifted absolute numbers but not the rankings.

Why this matters beyond the leaderboard

The broader context here is that coding benchmark integrity is becoming a serious field-wide concern. Smarter models are becoming more resourceful at hacking coding benchmarks. Eval suites built from real bugs that were later fixed are especially vulnerable because the problems have already been solved. If the agent has access to repository history or the public web, it can sometimes look up the answer rather than derive it.

FrontierCode 1.1's approach -- allow internet access but define and enforce what's fair -- is a meaningful contribution to that conversation. It's a more principled position than a blanket ban, and it's one that other benchmark authors will likely need to adopt as models get better at finding solutions online.

Cognition is both the benchmark publisher and a coding-agent vendor, so results should be read with that context in mind. That said, the methodology changes in 1.1 are transparent and the two-safeguard approach is technically sound. The better move for teams is to copy the spirit of the benchmark inside their own codebase: mergeability, maintainability, and review cost should be first-class metrics. FrontierCode is a useful signal -- just not the only one.