Artificial Analysis' FrontierCode 2.0 Drops Coding Agent Scores by 16 Points

Artificial Analysis, an independent AI evaluation firm, has released version 2.1 of its Coding Agent Index. The update replaces the previous FrontierCode component with Cognition’s harder FrontierCode 2.0 benchmark and refreshes Terminal-Bench to version 4.1. Most tested configurations lost points, and the new benchmark mix breaks direct score comparisons with earlier Index versions.

The top score of 58 belongs to four configurations:

• Claude Code with Claude Fable 5.2 at high reasoning effort
• Claude Code with Claude Fable 5.2 at extra-high, or xhigh, effort
• Claude Code with Claude Opus 5.5 at max effort
• Devin Fusion CLI with Claude Opus 5.5 Max and GPT-6 Luna High

Claude Code with Claude Sonnet 5.5 Max follows at 56. Gemini 4 Carbon at high effort scores 55 in Antigravity CLI.

A stricter merge bar drives the drop

The Index averages results from three tests and reports the result on a 100-point scale:

• FrontierCode-AA 2.0 evaluates repository patches against criteria that could block a merge.
• Terminal-Bench 4.1 measures an agent’s ability to complete tasks in a command-line environment.
• SWE-Atlas-QnA-AA tests software-engineering question answering.

FrontierCode 2.0 caused most of the movement. All 23 configurations evaluated on both versions scored lower on the new test. Their mean FrontierCode score fell 16 percentage points, from 45% to 29%, while overall Index scores declined by about five points on average.

Cognition redesigned FrontierCode 2.0 to judge whether a repository maintainer would accept an agent’s patch. The update includes:

• Tasks from 55% more repositories, including security fixes and performance optimizations
• 43% more blocking criteria per task
• Reference solutions that modify 67% more files and 89% more lines in codebases that are 25% larger
• Three times the previous share of thoroughness criteria

Thoroughness criteria penalize patches that fix the reported case while missing sibling code paths, related defects, or implied edge cases. Across the four frontier models Cognition analyzed, thoroughness produced the highest blocker failure rate at every reasoning level, exceeding failures on core behavior.

More reasoning can over-expand a patch

Higher reasoning settings sometimes reduce FrontierCode scores because agents write more code, touch more files, and create more opportunities to exceed the requested scope. Cognition documented one GPT-6 Astra task involving a race condition in the PostHog Android SDK. The low-effort run produced a minimal patch, while the max-effort run deleted a public API class and broke backward compatibility for downstream applications.

Claude Fable 5.2 follows a similar pattern in the new Index. Its max setting scores 56, two points below high and xhigh. At max effort, 16% of its FrontierCode attempts and 13% of its SWE-Atlas-QnA-AA attempts reached their time limits. The corresponding rates at xhigh were 1% and 0%.

Fable’s max setting also generates about three times as many output tokens per task as xhigh, increasing cost without improving the aggregate score.

Equal scores come with a 30-fold cost gap

Among configurations scoring at least 48, estimated pay-per-token API cost varies by roughly 30-fold. The following selected results show how reasoning effort changes the performance and cost of the same model:

Fable at high effort matches xhigh’s score for less than half the cost and exceeds max by two points at about one-sixth the price. Codex with GPT-6.1 Sol gives up six points against the leaders but costs $1.51 per task. Codex with GPT-6.1 Terra Max was the only tested configuration to remain within one point of its previous Index result.

Terminal-Bench barely moves

Terminal-Bench 4.0 includes 66 tasks for testing agentic terminal use. The revisions aim to improve grading reliability and reduce cases where an agent earns credit by exploiting the checker. Across the 23 configurations tested on both versions, every score moved by less than three points.

Kimi Code CLI with Kimi K3.1 Pro and Devin Fusion CLI each score 68% on the terminal benchmark. Claude Code with Claude Sonnet 5.5 Max reaches 66%, while Claude Opus 5.5 Max and Gemini 4 Carbon in Antigravity CLI each score 65%.

Fallback routing changes the comparison

Claude Code and Devin Fusion CLI automatically route an attempt to a secondary model after some safety refusals. The refusal remains recorded, while the fallback can still complete the task and preserve the benchmark result.

Fallback models completed 9% of attempts for Devin Fusion CLI and 9% for Claude Code with Claude Opus 5.5 Max. The rates were 6% for Claude Fable 5.2 Xhigh and 5% for Claude Sonnet 5.5 Max. Codex blocked about 1% of attempts with GPT-6.1 Sol Max and GPT-6 Astra Max.

Cross-harness scores therefore measure both model performance and orchestration policy, including refusal handling, fallback routing, and timeout behavior.

The tie conceals different strengths

Claude Fable 5.2 Xhigh scores 49% on FrontierCode-AA 2.0, five points above Claude Opus 5.5 Max. Opus leads Fable by four points on SWE-Atlas-QnA-AA, 66% to 62%, while their Terminal-Bench results are within one point.

Devin Fusion CLI reaches the same 58-point Index score through a different profile: 44% on FrontierCode, 68% on Terminal-Bench, and 63% on SWE-Atlas-QnA-AA.

A workload-based shortlist

• General repository work: Claude Fable 5.2 at high effort combines an Index-leading score of 58 with a reported cost of $7.51 per task.
• Terminal-heavy workflows: Kimi K3.1 Pro in Kimi Code CLI and Devin Fusion CLI share the highest Terminal-Bench score at 68%.
• Lower-cost deployments: Codex with GPT-6.1 Sol Max scores 52 at $1.51 per task.
• Reasoning-effort tuning: Claude Fable 5.2 Xhigh outperforms max while using fewer tokens and encountering fewer timeouts.

The cost figures estimate token charges at pay-per-token API rates. Subscription plans, private discounts, latency requirements, tool permissions, and repository-specific tasks can change the production trade-offs, so teams still need evaluations based on their own code and workflows.

Version 2.1 shifts more weight toward maintainer acceptance, including complete fixes, compatibility, and disciplined scope. The 16-point average decline on FrontierCode 2.0 shows that these requirements continue to separate leading coding-agent configurations.