FrontierCode 1.1 rewards code maintainers might merge

Cognition’s FrontierCode 1.1 evaluates coding agents on whether their patches fit a real repository’s requirements, conventions, and scope. Artificial Analysis has begun publishing results for frontier models using its open-source Stirrup harness, while the benchmark’s 100-task Main set remains private.

The leaderboard differs from rankings built primarily around unit-test completion. Claude Opus 5.5, using the reported “max with fallback” configuration, leads with an aggregate score of 52%. Nine models from six labs fall between 43% and 52%, and nine tasks were not solved by any evaluated model in any attempt.

The 52% result is an aggregate benchmark score, rather than a direct claim that the model completed exactly 52 of 100 tasks. Attempts that miss any blocking requirement receive zero; passing attempts earn weighted scores between 70% and 100% based on the remaining rubric items.

A maintainer-shaped quality bar

Many coding benchmarks focus on whether a patch passes a fixed test suite. FrontierCode also measures whether the change stays within scope, follows repository conventions, includes useful tests, and fits the existing architecture. These criteria target a recurring weakness in coding agents: a functionally correct patch can still create unnecessary files, alter unrelated code, or introduce an implementation that maintainers would reject.

Cognition worked with more than 20 open-source maintainers to construct tasks from repositories they maintain. According to Cognition, each task required more than 40 hours of development and review, including the definition of repository-specific criteria for a mergeable change.

How patches earn points

The grading pipeline combines executable tests with structural checks and model-based review:

• Reverse-classical tests run an agent’s new tests against the original broken code. Those tests must fail there, showing that they detect the targeted defect.
• Scope checks enforce file boundaries, diff-size limits, and semantic locality, reducing the chance that a broad rewrite passes a narrowly defined task.
• Adaptive classical grading uses an LLM to adjust reference tests for valid implementations with different names or interfaces, so superficial differences do not automatically cause failure.
• Prompt-based rubrics assess qualities such as readability, style, and architectural fit that are difficult to encode as deterministic tests.

Every rubric item is classified as either blocking or non-blocking. An attempt must satisfy all blockers before it can receive credit. Its final task score then reflects the weighted share of all satisfied criteria; otherwise, the score is zero.

A crowded leaderboard

Claude Opus 5.5 leads at 52%, followed closely enough that model choice depends on cost, latency, deployment constraints, and tolerance for failed tasks. MiMo-V2.6-Pro records the strongest open-weights result at 45%, while DeepSeek V4.1 Flash and Kimi K3 each reach 44%. Those results place the leading open-weights systems seven to eight points behind the top score.

Open weights can support self-hosting and deeper deployment control, although the term does not guarantee an open-source license, unrestricted commercial use, or access to training data. Teams evaluating these models should review each license and infrastructure requirement separately.

Cost changes the choice

Artificial Analysis reports average task costs ranging from $0.09 for GPT-6 Luna to $12.00 for Claude Fable 5.1. Four configurations sit on the score-cost Pareto frontier, meaning no other measured option is both cheaper and higher-scoring:

GPT-6.1 Sol matches Claude Fable 5.1’s 49% score at $0.81 per task, less than one-tenth of Fable’s reported $12.00 average. Opus 5.5 adds three percentage points over Sol while costing about 12.5 times as much per task in this run.

Token use accounts for part of that spread. GPT-6 Astra and GPT-6.1 Sol average 27,000 and 33,000 output tokens per task, respectively. Opus 5.5 averages 157,000, while Sonnet 5.5 reaches 236,000. DeepSeek V4.1 Flash completes a task in about seven minutes on average, compared with roughly 17 minutes for Opus 5.5.

These costs and completion times apply to the evaluated configurations, prompts, provider prices, and harness behavior. Production expenses can change with repository size, retry policy, context caching, tool calls, and inference provider.

Why familiar rankings move

FrontierCode’s ranking diverges from SWE-Bench because its grading criteria cover more than fixed-test completion. METR experiments cited by Cognition found that benchmark-successful patches can still fail human maintainers’ acceptance criteria. Cognition reports 81% fewer classification errors than other leading benchmarks in its validation, attributing the reduction to repository-specific rubrics and multi-stage review.

The prompts also provide less implementation guidance. Task descriptions average about one-third the length of SWE-Bench Pro prompts, requiring agents to infer more of the maintainer’s intent from the repository and issue context. This design tests planning and codebase navigation alongside implementation.

Choosing a model from the results

• For cost-sensitive, high-volume work: GPT-6.1 Sol offers a 49% score at an average of $0.81 per task.
• For the highest measured score: Claude Opus 5.5 reaches 52%, with substantially higher token use and cost.
• For tighter latency or token budgets: the reported GPT-6 configurations use fewer output tokens, while DeepSeek V4.1 Flash posts the shortest cited completion time.
• For open-weights deployment: MiMo-V2.6-Pro leads that category at 45%, subject to its licensing and infrastructure requirements.

FrontierCode offers a useful proxy for maintainability by combining tests, scope controls, and qualitative review. Its private task set can reduce exposure to benchmark contamination, while the open harness makes the execution layer inspectable. The private tasks and LLM-assisted grading also limit independent auditing, so teams should pair the leaderboard with trials on their own repositories.

A top aggregate score of 52% leaves substantial room for failed blockers and incomplete solutions. Developers adopting coding agents still need repository-specific evaluation, review gates, and human approval for changes that affect production systems.