Epoch's MirrorCode Shows Claude Fable 5 Crushing GPT-5.6 Sol 64% to 20%

MirrorCode hands an AI agent a compiled binary it cannot read, some documentation, and a set of visible test cases, then asks it to rebuild the entire program from scratch in a language of its choosing, passing a hidden test suite it will never see during development. Epoch AI just updated the leaderboard with results for Claude Fable 5 and GPT-5.6 Sol, and the gap between them is striking.

What MirrorCode actually measures

MirrorCode is a long-horizon software engineering benchmark that evaluates autonomous coding by requiring AI agents to fully reimplement existing CLI programs. No bug fixes, no function stubs, no isolated snippets. Agents must manage architecture, design, and complex logic across extended sessions. The benchmark was co-developed with METR, which also provided grant funding.

What separates MirrorCode from most benchmarks is the inference budget. Many existing software engineering benchmarks cap spending at $1–$10, even for tasks that would take a human weeks. On MirrorCode, a single run can cost thousands of dollars and stretch across multiple days.

Each task follows the same structure:

• The agent gets execute-only access to the compiled binary, text documentation, and visible end-to-end test cases. No source code, no internet.
• Solutions must match the original program's output exactly on end-to-end tests, including held-out cases.
• A solve requires passing 100% of both visible and hidden tests. No partial credit.

The 25 target programs span Unix utilities, data serialization, query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Agents can write their solutions in Python, C, Rust, Go, OCaml, or Ada.

The leaderboard numbers

The leaderboard configuration is specific and worth understanding before reading the scores. Models are tested on 15 Medium and Large programs, in two implementation languages each, with a budget of 10 billion tokens per attempt. Each task runs three times, giving 30 tasks total. These scores are not directly comparable to the original paper results, which used a different configuration.

Claude Fable 5's 64% is a large jump over the prior best. Leading models from eight months ago would have scored around 30% and were limited to simpler tasks. The benchmark was designed to resist saturation, and it still is, but the ceiling is rising fast.

Two tasks nobody had solved before

Claude Fable 5 is the first model to solve the C preprocessor and Pkl tasks in at least one run. Both had been documented failure cases for earlier generations.

The Pkl task illustrates what "hard" means here. Pkl is a full interpreter for a lazily evaluated language designed to replace JSON and YAML in complex configuration workflows. Implementing it correctly requires handling lazy evaluation (where properties are computed when accessed, not when defined), parsing from a behavioral specification rather than readable source code, and passing a test suite drawn partly from real-world usage examples.

Earlier models had a specific, documented failure mode on this task. Despite the Pkl specification repeatedly emphasizing lazy evaluation, agents chose eager evaluation in their first draft and spent the rest of the session patching around that decision. Even when the model identified the problem midway through, it kept eager evaluation anyway. That is a classic failure of long-horizon planning: correct local reasoning, wrong architectural commitment. Fable 5 cleared it, which represents a qualitatively different behavior from its predecessors.

Ada performs better than expected

In the standard leaderboard configuration, models must write half their solutions in Ada, a niche safety-critical language with a syntax and type system far removed from the languages that dominate training data. The expected result is a significant performance drop.

It does not happen. Frontier models perform almost as well in Ada as in Go, a far more popular language. The result suggests these models have learned generalized programming skills rather than syntax pattern-matching. There is a small effect on token usage: successful Python solutions tend to use fewer tokens than average, while successful Ada solutions tend to use more. Solve rates stay close regardless.

What this means for real engineering work

The skills required to succeed at long-horizon tasks, including sustained coherent planning across thousands of lines, detecting and correcting deep architectural mistakes, and iterating over days without human check-ins, are qualitatively different from single-turn code generation. MirrorCode results show AI can now sustain that kind of work on a complex codebase with no human intervention.

The benchmark has already produced concrete data points. Claude Opus 4.7 substantially reimplemented gotree, a bioinformatics toolkit with roughly 16,000 lines of Go and 40+ commands. Epoch AI estimates the same task would take a human engineer without AI assistance 2–17 weeks. Opus 4.7 solved it in 14 hours, passing 2,000 of 2,001 tests, at a cost of $251.

The practical use cases this unlocks include:

• Porting legacy codebases to new languages or platforms without access to the original source, such as from a compiled binary or proprietary system
• Reimplementing open-source tools in a different language for performance, compliance, or dependency reasons
• Autonomous agent pipelines that need to build or extend tooling without human checkpoints at every step
• AI safety evaluation, since MirrorCode also tracks how close AI is to autonomously conducting AI R&D tasks

Open benchmark, public code

Epoch AI releases their scaffold and 22 of the 25 MirrorCode target programs as open source on GitHub, covering 132 task instances across the six supported languages. Three targets are held back as a private test set to prevent overfitting. The full paper covers benchmark construction, contamination analysis, and detailed per-task results.

Eight of the 25 targets have never been perfectly solved, and even the strongest models struggled with the largest, most complex projects. Epoch AI says more model scores will be added to the leaderboard shortly. Eight months ago, MirrorCode was a research curiosity. At 64%, it is becoming a real measure of how much of a software project an AI can own end-to-end.