FrontierCS 2 Challenges AI Agents to Rebuild Missing Research Code
A new benchmark strips the key contribution from 48 research papers and asks AI agents to rebuild it well enough to beat the authors' own code.
- FrontierCS 2 benchmarks 12 LLM agents on 48 research tasks built by deleting a paper's contribution from its released code
- Agents must rebuild the deleted code and beat the authors' version on every metric and held-out workload to pass
- Claude Opus 5.5 leads with 58% pass rate; every other evaluated agent passes under half its runs
- Sixteen tasks saw zero passes from any of the 12 agents, with the median task matching the strongest agent's capability
- New FrontierCS Capabilities Index puts agents and tasks on a shared item-response-theory scale
- Full evaluation harness is open source at github.com/FrontierCS/FrontierCS-2, with 100+ private tasks held in reserve
FrontierCS 2 evaluates whether coding agents can improve research software by removing each paper’s key implementation and asking an agent to replace it. Across 48 published papers in 13 domains, the strongest tested setup, Claude Opus 5.5 in Claude Code, passed 58% of its runs. Sixteen tasks defeated every agent.
The benchmark targets a common research workflow: inherit a working repository, change one component, test it on established workloads, and compare the result with the published implementation. Earlier open-ended benchmarks often provide a problem statement and grade a single generated program. That setup omits repository navigation, dependency constraints, iterative testing, and regressions across multiple metrics.
| Design choice | FrontierCS 2 approach |
|---|---|
| Scope | 48 papers across 13 research domains |
| Systems tested | 12 model-and-harness combinations |
| Time limit | Five hours per run |
| Development feedback | A score and a sanitized count of metrics meeting their thresholds |
| Final evaluation | Held-out workloads unavailable during development |
| Pass criterion | Beat the human reference on every metric and workload |
Research code with its core removed
The authors construct each task by locating the code that implements a paper’s main contribution, deleting it, and preserving the surrounding repository. The deleted implementation becomes the private human reference that the agent must beat.
In the SVG-EAR task, derived from a sparse video-attention paper, the benchmark removes the error-aware attention class and three of its routines. Function declarations, callers, dependencies, and other scaffolding remain, leaving the agent to infer the missing behavior and develop a competitive replacement.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
