Epoch's MirrorCode Shows Claude Fable 5 Crushing GPT-5.6 Sol 64% to 20%
Epoch AI's MirrorCode leaderboard puts Claude Fable 5 at 64% and GPT-5.6 Sol at 20% on full-program reimplementation tasks that can run for weeks and cost thousands of dollars per attempt.
- Claude Fable 5 tops the MirrorCode leaderboard at 64% solve rate; GPT-5.6 Sol follows at 20%, GPT-5.4 at 16%, GPT-5.5 at 10%.
- MirrorCode requires AI to reimplement entire CLI programs from a compiled binary with no source code access, passing hidden end-to-end tests at 100%.
- Claude Fable 5 is the first model to solve the C preprocessor and Pkl tasks -- both previously documented failure cases for earlier model generations.
- Frontier models perform nearly as well writing in obscure Ada as in popular Go, suggesting generalized programming skill rather than syntax memorization.
- The leaderboard uses a 10-billion-token budget per attempt; one of the hardest tasks cost $2,600 for a single run and ran for 19 days.
- 22 of 25 target programs and the full scaffold are open-source on GitHub; the paper is on arXiv.
MirrorCode hands an AI agent a compiled binary it cannot read, some documentation, and a set of visible test cases, then asks it to rebuild the entire program from scratch in a language of its choosing, passing a hidden test suite it will never see during development. Epoch AI just updated the leaderboard with results for Claude Fable 5 and GPT-5.6 Sol, and the gap between them is striking.
What MirrorCode actually measures
MirrorCode is a long-horizon software engineering benchmark that evaluates autonomous coding by requiring AI agents to fully reimplement existing CLI programs. No bug fixes, no function stubs, no isolated snippets. Agents must manage architecture, design, and complex logic across extended sessions. The benchmark was co-developed with METR, which also provided grant funding.
What separates MirrorCode from most benchmarks is the inference budget. Many existing software engineering benchmarks cap spending at $1–$10, even for tasks that would take a human weeks. On MirrorCode, a single run can cost thousands of dollars and stretch across multiple days.
Each task follows the same structure:
- The agent gets execute-only access to the compiled binary, text documentation, and visible end-to-end test cases. No source code, no internet.
- Solutions must match the original program's output exactly on end-to-end tests, including held-out cases.
- A solve requires passing 100% of both visible and hidden tests. No partial credit.
The 25 target programs span Unix utilities, data serialization, query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Agents can write their solutions in Python, C, Rust, Go, OCaml, or Ada.