Google's Gemini 4 Carbon Scores 40% on Cognition's Toughest Coding Benchmark
Cognition evaluated Google's new Gemini 4 Carbon and Argon on FrontierCode 2.0, a mergeability benchmark where even the top model scores just 45%.
- Gemini 4 Carbon scores 40% at $4.04/task on FrontierCode 2.0, near state of the art.
- Gemini 4 Argon trails at 26.3% and costs more at $6.41 per task.
- Claude Opus 5.5 leads the benchmark at 45.5%, showing it remains far from saturated.
- FrontierCode 2.0 adds 132 tasks with larger scope and stricter mergeability grading.
- Thoroughness blockers tripled, making unstated-constraint failures the top reason models lose points.
- Higher reasoning effort can reduce scores because models expand scope and break APIs.
Cognition has published FrontierCode results for Google’s Gemini 4 Carbon and Gemini 4 Argon coding models. Carbon scored 40% at a reported cost of $4.04 per task, placing behind three leading models. Argon scored 26.3% in FrontierCode 2.0.
The 132-task benchmark evaluates whether agent-generated pull requests satisfy maintainer-written requirements for correctness, design, scope, and cleanup. Its criteria aim to reflect changes that an open-source maintainer could merge, which helps explain why scores remain low even for leading models.
Carbon Reaches the Leading Group
Carbon trails Claude Opus 5.5, GPT-6 Astra, and Claude Sonnet 5.5. Argon sits below Claude Haiku 5.5 and Grok 4.7, though it outperforms Muse Wonder 1.0. Opus 5.5 leads the full benchmark with 45.5%, leaving more than half of the available score unearned.
Cognition’s version comparison shows substantial declines from FrontierCode 1.1 to 2.0:
| Model | FrontierCode 1.1 | FrontierCode 2.0 | Cost per 2.0 task |
|---|---|---|---|
| Claude Opus 5.5 | 54.6% | 45.5% | N/A |
| GPT-6 Astra | 53.3% | 42.8% | N/A |
| Gemini 4 Carbon | N/A | 40.0% | $4.04 |
| Grok 4.7 | 47.6% | 28.6% | N/A |
| Gemini 4 Argon | 47.5% | 26.3% | $6.41 |
| SWE-2 | 50.0% | 23.9% | N/A |
| Kimi K3 | 44.2% | 17.7% | N/A |
The Rubric Gets Stricter
Cognition rebuilt FrontierCode 2.0 to distinguish models as older coding benchmarks approached saturation. The new version raises the difficulty in three ways:
- Tasks cover more repositories and categories, including security patches and performance optimizations.
- Grading applies broader correctness and design requirements, with greater weight on complete implementations.
- Reference solutions modify 67% more files and 89% more lines on average, across codebases that are 25% larger.
FrontierCode labels inferred correctness requirements as “thoroughness” criteria. These checks cover behavior that an agent should derive from the prompt and surrounding code, such as resource cleanup, lifecycle handling, and compatibility with existing interfaces.
The number of thoroughness criteria tripled from version 1.1, and the category now causes more failures than any other. Models frequently implement the requested behavior while missing consequences elsewhere in the repository.
Implicit Constraints Break Otherwise Plausible Patches
Cognition developed the rubric around problems observed while deploying Devin. Coding agents often misread intent, overlook unstated constraints, or introduce maintenance costs that require human intervention.
A task from Mock Service Worker illustrates the failure mode. The prompt asks the agent to fix a bug in which resetHandlers() fails to restart a generator. Claude Opus 5.5 removes the old generator reference, which restores the requested behavior, but leaves the generator open. Any cleanup placed in the user’s finally block never runs. All five evaluated attempts failed the task.
More Reasoning Can Widen the Patch
FrontierCode rewards satisfaction of its criteria, so additional code can reduce a score when a model expands the patch beyond the requested scope. Higher reasoning settings also tend to increase the number of files and lines changed.
In Cognition’s PostHog example, GPT-6 Astra passes at low reasoning effort with a focused modification. At maximum effort, it deletes a public API class and introduces a breaking change. The result suggests that teams should test reasoning settings alongside model versions instead of assuming the largest inference budget produces the safest patch.
How Developers Can Use the Results
For autonomous coding work, the useful comparison combines rubric performance with evaluation cost. Carbon scored 13.7 percentage points above Argon while costing $2.37 less per task, making it the stronger Gemini option under Cognition’s benchmark configuration. Argon also falls behind lower-cost Haiku and Grok alternatives reported on the leaderboard.
The 45.5% leading score supports continued use of repository-specific tests, review gates, and rollback paths for agent-generated changes. Review criteria should cover lifecycle cleanup, public API compatibility, edge cases implied by nearby code, and unnecessary expansion of patch scope.
Production results will also vary with the repository, agent harness, tools, prompts, and pricing assumptions. Teams evaluating coding agents can reproduce the benchmark’s core lesson by testing several reasoning levels on representative tasks and measuring accepted changes, review time, regressions, and total cost.
Comments
