Epoch's MirrorCode Shows Claude Fable 5 Crushing GPT-5.6 Sol 64% to 20%
Epoch AI's MirrorCode leaderboard puts Claude Fable 5 at 64% and GPT-5.6 Sol at 20% on full-program reimplementation tasks that can run for weeks and cost thousands of dollars per attempt.
- Claude Fable 5 tops the MirrorCode leaderboard at 64% solve rate; GPT-5.6 Sol follows at 20%, GPT-5.4 at 16%, GPT-5.5 at 10%.
- MirrorCode requires AI to reimplement entire CLI programs from a compiled binary with no source code access, passing hidden end-to-end tests at 100%.
- Claude Fable 5 is the first model to solve the C preprocessor and Pkl tasks -- both previously documented failure cases for earlier model generations.
- Frontier models perform nearly as well writing in obscure Ada as in popular Go, suggesting generalized programming skill rather than syntax memorization.
- The leaderboard uses a 10-billion-token budget per attempt; one of the hardest tasks cost $2,600 for a single run and ran for 19 days.
- 22 of 25 target programs and the full scaffold are open-source on GitHub; the paper is on arXiv.
MirrorCode is not your typical coding benchmark. There are no bug fixes, no function stubs to complete, no isolated snippets. Instead, an AI agent is handed a compiled binary it cannot read, some documentation, and a set of visible test cases -- then told to rebuild the entire program from scratch, in a language it chooses, passing a hidden test suite it will never see during development. Epoch AI just updated the leaderboard with results for Claude Fable 5 and GPT-5.6 Sol, and the gap between them is striking.
What MirrorCode actually measures
MirrorCode is a long-horizon software engineering benchmark that evaluates autonomous coding by requiring AI agents to fully reimplement existing CLI programs. Unlike traditional benchmarks that focus on short code snippets or bug fixes, MirrorCode tasks involve building entire programs from scratch, requiring agents to manage architecture, design, and complex logic over extended periods.
The benchmark was co-developed with METR and supported by a grant from METR. What makes it unusual is the inference budget. Epoch AI provides a large enough inference budget to make a serious attempt at MirrorCode tasks. Many existing software engineering benchmarks limit inference spending to around $1-10, even when the task would take weeks for a human to complete. On MirrorCode, a single run can cost thousands of dollars and run for days.
The setup for each task looks like this:
- The AI has no access to the original source code or the internet. It is given execute-only access to the original compiled binary (a "black box" it can probe), text documentation of the program, and "visible" end-to-end test cases to clarify the exact scope of what needs to be built.
- AI-generated solutions must match the original program's output exactly on end-to-end tests, including held-out tests.
- A solve requires passing 100% of both visible and hidden tests -- no partial credit.