Cognition's FrontierCode 1.1 Asks If AI Code Is Actually Mergeable

FrontierCode 1.1 is a refinement of Cognition's month-old benchmark that asks a harder question than most coding evals: not "does this code run?" but "would a maintainer actually merge this pull request?" The update tightens the methodology on two fronts that were quietly distorting scores -- what counts as fair internet use, and which grading criteria were too strict to be meaningful.

Why this benchmark exists in the first place

Today's coding benchmarks have established that models can write correct code. But as AI-generated code becomes the dominant path to production, correctness is now table stakes. The real question is whether models can write good code.

FrontierCode is a software-engineering benchmark that evaluates whether coding agents produce mergeable, production-quality pull requests, scoring correctness, tests, scope, style, and maintainability through maintainer-authored rubrics. Cognition consulted more than 20 leading open-source maintainers spanning 36 flagship repositories to develop the benchmark tasks, with each task requiring over 40 hours of expert contribution.

The benchmark grades along five axes:

• Behavioral correctness -- does the patch actually solve the problem?
• Regression safety -- does it break anything that was already working?
• Scope discipline -- does the PR touch only what it should, or does it sprawl into unrelated files?
• Test quality -- are the new tests actually validating the right behavior?
• Style and codebase standards -- does the code look like it belongs in this repo?

The benchmark contamination problem everyone is dealing with

FrontierCode tasks are drawn from real pull requests in open-source repos. That realism is a feature -- but it creates a vulnerability. These suites draw tasks from real, already-fixed open-source bugs. Because each bug was fixed, the answer often exists online. A capable agent can search for it rather than reason through the code.