Epoch's MirrorCode Benchmark Shows Claude Opus 4.7 Rebuilding Software Autonomously in 14 Hours

How long can an AI model code on its own before it needs a human to step in? That question has been frustratingly hard to answer with existing benchmarks, which mostly test bug fixes and small feature additions. MirrorCode, a new benchmark from Epoch AI co-developed with METR, takes a radically different approach: give an AI model a compiled binary it cannot read, documentation, and a test suite, then ask it to rebuild the entire program from scratch.

The results are striking. Claude Opus 4.7 achieved a headline score of 56% across 25 target programs, and on one task, reimplementing gotree (a bioinformatics toolkit with 16,000 lines of Go and 40+ commands), it passed 99.95% of tests. The team estimates that same task would take a human engineer 2 to 17 weeks. Opus 4.7 finished in 14 hours for $251.

Why existing SWE benchmarks fall short

AI models are increasingly capable at autonomous coding, and several notable software engineering benchmarks have seen rapid progress. However, these usually measure fairly short coding tasks; for example, only about 100 of the 731 SWE-bench Pro tasks involve diffs larger than 100 lines. Meanwhile, recent demos of AI coding, such as developing a new C compiler or a new browser, are impressive but hard to evaluate. The completeness of the resulting software is debatable, and the extent of human guidance is unclear, making it difficult to use these as a proxy for autonomous AI coding.

There is also a deeper structural problem: current benchmarks fall short on two dimensions: horizon and verifier strength. Dominant public benchmarks measure agent performance within minute-scale; even some of the most challenging have most tasks resolved within an hour by top agents. Spending $5 on a task that would take a human weeks is not a fair test of AI's ceiling.

The MirrorCode setup

MirrorCode addresses these problems by constructing a long-horizon coding benchmark based on existing software projects. Each task consists of a command-line program that an agent is tasked to reimplement exactly. The AI agent is given execute-only access to the original program and a set of visible test cases, but does not have access to the original source code. Think of it as: here is the black box, here is the manual, now build the same thing.

The benchmark spans a deliberately broad set of domains:

• Unix utilities (e.g., cal, choose)
• Bioinformatics toolkits (e.g., gotree)
• Interpreters and configuration languages (e.g., Pkl)
• Data serialization and query tools
• Static analysis, cryptography, and compression

Each target program is tested against hundreds to thousands of end-to-end test cases, including a set of held-out tests the model never sees during development. MirrorCode provides a large enough inference budget to make a serious attempt at tasks. Many existing software engineering benchmarks limit inference spending to around $1–10, even when the task would take weeks for a human to complete. For example, one of the largest MirrorCode tasks cost $2,600 for a single run and involved AI working for 19 days without human intervention.

How it actually works, step by step

The agent runs inside a sandboxed Docker container. Here is the flow for a single task:

1. The agent receives the compiled binary (execute-only), documentation, and a set of visible test cases.
2. It can run the binary with arbitrary inputs to probe its behavior, acting as a black-box oracle.
3. It writes its reimplementation in a chosen language (Python, C, Rust, Go, OCaml, or Ada).
4. It iterates, running visible tests and refining its solution.
5. When it submits, the solution is scored against both visible and held-out tests in a separate Docker container the agent cannot access.

The evaluation relies on a very particular setup: an existing program that produces the canonical output for a given input, and hence acts as a highly detailed, precise specification. Although this setup can arise in real-world reverse engineering and reimplementation, it is not how software is typically developed.

To prevent cheating, the team uses paired "dual" test cases. For example, a visible test might check the calendar output for February 1983. The hidden dual checks a different year. If the model hardcodes the answer, it fails the hidden version.

What the numbers actually show

However, MirrorCode is not fully solved. Claude Opus 4.7's headline score is only 56%, meaning there is significant room for further improvement. AI models are also improving rapidly over time: leading models from a year ago would have scored about 30%, and were limited to simpler programs such as a calendar utility.

The per-model breakdown on headline solve rate (passing 100% of tests):

On the gotree task specifically, the progression across four generations of Claude Opus tells a sharp story:

On 21 of 25 MirrorCode targets, AI models have at least once passed 99% of tests or more. Typically, outstanding test failures come from a handful of edge cases. The benchmark is not saturated, but it is also not a wall. Models make real, measurable progress.

The hidden story: what models get wrong

The qualitative analysis in the paper is where MirrorCode really earns its keep. The gotree case study reveals two failure modes that older models share and newer ones fix.

Premature submission. Opus 4.0 submitted with 92% of tests still failing, framing single-digit improvements as "great progress." Opus 4.1 hallucinated a time constraint that was never mentioned in the prompt and quit at 23% completion, having used less than 1% of its token budget. Opus 4.6 submitted only when all visible tests passed.

Wrong data structures. Newer models exhibit better judgment about when to submit, superior data structure selection (graph-based Edge objects versus generic trees), and sustained perseverance through complex problems. Older models stored branch lengths on child nodes rather than on edges, which caused cascading failures during tree topology operations like rerooting.

The hardest task in the suite, Pkl (Apple's configuration language, ~61,000 lines of Java/Kotlin), remains unsolved. The failure is architectural: Pkl requires lazy evaluation (properties are computed only when accessed), but every model chose eager evaluation in its first draft and spent the rest of its budget patching around that decision. The agent repeatedly diagnosed the correct fix, then declined to implement it, citing complexity. With 77% of its token budget still remaining, Opus 4.6 never rewrote the evaluator.

Three design innovations that matter

MirrorCode solves three problems that have plagued long-horizon SWE benchmarks:

• Fairness at scale. The benchmark provides a large enough inference budget to make a serious attempt. One of the largest runs lasted 19 days and cost $2,600. The task is hard, but the model is not artificially starved of compute.
• Cheat resistance. Models are sandboxed with no internet access and no way to read the original source. Held-out dual tests catch hardcoding. AI models can autonomously reimplement complex existing software without access to the original program's source code, provided there is a detailed, checkable specification.
• Language agnosticism. Models can implement in Python, C, Rust, Go, OCaml, or Ada. The benchmark found little sign of programming language affecting performance, even in obscure languages like Ada.

What it is and is not good for

MirrorCode is most useful as a signal for tasks where a precise, testable spec exists: reimplementations, protocol implementations, format converters, CLI tool clones, and any project where the desired behavior can be fully captured in end-to-end tests. When given precise specifications and test suites, current AI models can sustain complex architectural decision-making across thousands of lines of code without human intervention.

It is a poor proxy for the messier parts of software engineering. It is not common for real software to be developed against a precise, programmatically checkable specification, and it is unclear how these findings translate to real software development. Ambiguous requirements, evolving stakeholder needs, and judgment calls about tradeoffs are not tested here.

There is also the memorization question. Because the target programs are open source, models may have seen them during pretraining. The team ran a memorization screen (comparing model-generated code to original source using character-level similarity) and excluded programs that showed clear evidence of copying. The included targets clustered near the uncontaminated baseline of ~0.34 similarity. But partial memorization of algorithmic approaches cannot be ruled out.

What is now possible

For some tasks, AI is already as good as a full-time sophisticated employee. Imagine giving a talented software programmer a CLI interface to a complicated program and asking them to write the underlying program without seeing its source code. Only a fraction of humans could do it if the program was quite sophisticated, and the ones that could would likely spend many days working on it. The fact AI can do this task autonomously is remarkable.

Existing results suggest that AI's ability to autonomously complete tasks may be correlated with the presence of a feedback signal. Consider recent AI progress on research engineering benchmarks, CUDA kernels, and other end-to-end measurable research engineering tasks. MirrorCode gives the field a rigorous, reproducible way to track that frontier.

The scaffold and 22 of 25 target programs are open source, with three held out as a private test set. The full paper includes the leaderboard, detailed methodology, and the qualitative case studies on gotree and Pkl that are worth reading in full if you are building or evaluating long-horizon coding agents.