GitHub Copilot Matches Claude Code While Burning Fewer Tokens
GitHub benchmarks its own agentic harness against Claude Code and Codex CLI, showing token efficiency gains without sacrificing task resolution across 20+ models.

- Harness vs. model: GitHub benchmarks its Copilot agentic harness against Claude Code and Codex CLI, showing task resolution parity with fewer tokens across most configurations.
- Five benchmarks, four models: Tests ran on SWE-bench Verified, SWE-bench Pro, SkillsBench, TerminalBench, and Win-Hill using Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.4, and GPT-5.5.
- 20+ models, one harness: Copilot CLI supports GPT, Claude, Gemini, and MAI families plus bring-your-own-key, enabling per-task model routing.
- Rubber Duck: An experimental cross-model critic closes 74.7% of the performance gap between Claude Sonnet and Opus by having a GPT model review Claude's work (and vice versa).
- Token-based billing: Copilot moved to AI Credits (1 credit = $0.01) on June 1, 2026; fewer tokens per task directly lowers cost per completed task.
- Field implication: The same model scores differently in different harnesses, making orchestration layer quality a primary evaluation criterion alongside raw model capability.
There is a quiet assumption baked into most AI coding benchmarks: the model is what matters. GitHub's latest research challenges that directly. The company has published a detailed benchmark study showing that its agentic harness, the orchestration layer wrapping the model, can match the task resolution rates of model-vendor harnesses like Claude Code and Codex CLI, while consuming fewer tokens across most configurations.
The harness is the product
While the model provides the raw intelligence, the harness shapes how effectively that intelligence is applied. The GitHub Copilot agentic harness is a single shared component of the GitHub Copilot SDK, which powers the GitHub Copilot CLI, GitHub Copilot app, and Copilot code review, along with a wide variety of experiences across GitHub and Microsoft. The key architectural insight here is that improving the harness means every surface benefits, from the CLI to VS Code to Xcode, all at once.

A harness, in this context, is the scaffolding that turns a raw LLM into a coding agent: it assembles context, selects tools, manages the agent loop, handles memory, and feeds results back to the model. Two agents can call the same model and produce very different outcomes because they give the model different tools, different prompts, different memories, and different feedback. That gap is exactly what this study tries to measure.
How the benchmarks were run
GitHub compared the Copilot agentic harness against model-vendor harnesses (Claude Code and Codex CLI) across five benchmarks: SWE-bench Verified, SWE-bench Pro, SkillsBench, TerminalBench, and an internal Win-Hill benchmark, using four models: Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.4, and GPT-5.5.
Here is what each benchmark tests:
- SWE-bench Verified: 500 human-validated bug-fix tasks from real open-source Python repos. The industry standard for coding agents.
- SWE-bench Pro: Harder, multi-step engineering tasks requiring deeper reasoning and broader code changes.
- SkillsBench: Tests how effectively an agent uses and triggers skills to solve tasks.
- TerminalBench: Agent performance on terminal-based, command-line workflows.
- Win-Hill: An internal GitHub benchmark running tasks inside Windows containers, to validate cross-OS generalization.
GitHub compared Copilot CLI with Claude Code for Claude models and Codex CLI for GPT models, normalizing settings such as context window, reasoning effort, tool selection, and MCP servers. The methodology included five independent runs per configuration and controlled normalization of context windows and reasoning effort.
Token efficiency: the headline result
The headline is measured but important: GitHub says Copilot's harness reaches task resolution on par with leading model-vendor harnesses while often using fewer tokens. The token efficiency charts show Copilot CLI consistently at or below the token consumption of Claude Code and Codex CLI across most model-benchmark combinations.

The one notable exception: on SWE-bench Verified with GPT-5.4 and GPT-5.5, Copilot CLI used 7% and 4% more tokens respectively than Codex CLI. GitHub is transparent about this in the post, which adds credibility to the rest of the results.
Task resolution: effective parity
Task resolution rates for the GitHub Copilot agentic harness across these benchmarks are on-par with model-vendor harnesses when used with a fixed model and benchmark task. This ensures that the full potential of the underlying model is available, along with multi-model flexibility, token efficiency, and memory and context capabilities.
These results reflect effective parity, since the differences in either direction are within the variance due to the stochastic nature of the models, making the cross-harness performance on-par. The TerminalBench variance analysis makes this concrete: GitHub ran each agent-model combination at least five times and plotted resolution rate against cost per task, with error ellipses showing run-to-run spread. Copilot's markers sit within overlapping ellipses of their same-model competitors on nearly every configuration.

The real unlock: 20+ models, one harness
Matching vendor-native harnesses on benchmarks is table stakes. The more interesting story is what a model-agnostic harness enables that a vendor-native one cannot. The GitHub Copilot agentic harness supports 20+ frontier models across the GPT, Claude, Gemini, and MAI families, plus bring your own key for open-source and local models. You can choose the right model for the capability and cost profile of each task, or let Auto model selection choose for you, balancing task intent and model health to optimize token efficiency.
The TerminalBench variance chart illustrates the practical trade-off clearly: GPT models deliver strong resolution at the lowest cost, while Claude Opus reaches the highest resolution at a premium. A vendor-native harness forces you to pick one. Copilot lets you route by task.
This also unlocks a genuinely novel capability called Rubber Duck. A multi-model architecture unlocks harness-level capabilities a model-vendor harness simply cannot offer. Rubber Duck uses cross-model-family critique, where one model reviews another's work to improve outcomes beyond what any single model produces alone.
Rubber Duck: a second opinion from a different brain
Rubber Duck leverages a second model from a different AI family to act as an independent reviewer, assessing the agent's plans and work at the moments where feedback matters most. To catch different kinds of errors, a different perspective matters. Evaluations show that Claude Sonnet + Rubber Duck makes up 74.7% of the performance gap between Sonnet and Opus alone, achieving better results for tackling difficult multi-file and long-running tasks.
A key design feature of the rubber duck agent is that it deliberately runs on a different AI model from the one driving your session. GitHub Copilot CLI picks a critic model that contrasts with the current session model. For example, if you have chosen to use a Claude model for your session, the rubber duck agent may use a GPT model as the critic. The reasoning: the critic is less likely to share the same blind spots, biases, or failure modes as the model that produced the work.
Rubber Duck activates automatically at three checkpoints:
- After drafting a plan, before implementation, where catching a wrong assumption is cheapest.
- After a complex implementation, to catch edge cases a single model might miss.
- After writing tests, before running them, to catch gaps in coverage or flawed assertions.
Rubber Duck tends to help more with difficult problems, ones that span 3+ files and would normally take 70+ steps. On these problems, Sonnet + Rubber Duck scores 3.8% higher than the Sonnet baseline, and 4.8% higher on the hardest problems identified across three trials. You can also trigger it manually at any point with a natural language prompt like "rubber duck your plan," or via the /experimental slash command in the CLI.
Pricing and how to use it
Copilot Pro and Pro+ monthly subscriptions include monthly AI Credits aligned to their current subscription prices: Copilot Pro at $10/month and Copilot Pro+ at $39/month. GitHub AI Credits are how you pay for AI usage in GitHub Copilot, where every plan includes a monthly allowance and 1 AI credit equals $0.01 USD. You use credits when you chat with Copilot, work with agents, or use Copilot CLI, Spaces, and Spark. Code completions and next edit suggestions do not use credits and remain unlimited with every paid plan.
This matters directly for the token efficiency story. Usage is calculated based on token consumption, including input, output, and cached tokens, using the listed API rates for each model. A harness that uses fewer tokens per task is not just a benchmark win, it is a lower credit bill per completed task. The GitHub Copilot CLI is available on all paid plans, and the Copilot SDK lets teams build on the same harness directly.
What this means for the field
The most important story here is not that another model became smarter. It is that GitHub published a public argument for a different unit of competition: the agentic harness. For the past two years, every benchmark headline has been about model capability. This study reframes the question: given the same model, how much does the orchestration layer matter?
The lesson underneath is that the same model scores differently in different harnesses, which is exactly why the wrapper matters as much as the model. The practical implication for teams adopting AI coding agents is that evaluating tools by model alone is no longer sufficient. Resolution rate, cost per task, run-to-run variance, and model flexibility are all now first-class metrics. GitHub's companion post on context handling and model routing goes deeper on the specific optimizations driving these results.