Skip to main content
Back

OpenAI's frontier-evals Bundles Three Brutal Agent Benchmarks Into One Open Repo

OpenAI consolidates three of its toughest agent benchmarks into a single MIT-licensed repo covering paper replication, freelance engineering, and smart contract exploitation.

Repo, Full Access
  • OpenAI published frontier-evals, an MIT-licensed repo bundling PaperBench, SWE-Lancer, and EVMbench.
  • PaperBench tasks agents with replicating 20 ICML 2024 papers across 8,316 gradable sub-tasks.
  • Best agent hits only 21% replication on PaperBench; ML PhDs still beat every tested model.
  • SWE-Lancer covers 1,400+ Upwork tasks worth $1M, graded with triple-verified end-to-end tests.
  • EVMbench evaluates detect, patch, and exploit modes across 117 real smart contract vulnerabilities.
  • GPT-5.3-Codex scores 71% on EVMbench exploit mode, more than doubling GPT-5's 33.3%.

OpenAI has quietly consolidated three of its most demanding agent evaluations into a single public repository called frontier-evals. The suite bundles PaperBench, SWE-Lancer, and EVMbench under one MIT license, each with its own isolated project, dependencies, and reproduction instructions. If you want to stress-test a coding agent against something meaningfully harder than SWE-bench, this is now the one-stop shop.

Each benchmark targets a different flavor of long-horizon engineering work, and together they answer a question that leaderboard scores on toy tasks no longer can: can your agent actually finish an economically valuable piece of work end to end?

Three benchmarks, three very different pain points

PaperBench is the most ambitious of the three. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. Grading is not a vibe check. Rubrics hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria, and in total PaperBench contains 8,316 individually gradable items, co-developed with the original paper authors. An LLM-based judge handles scoring at scale, and a separate meta-benchmark validates the judge itself.

The headline result is humbling. The best-performing tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, achieves an average replication score of 21.0%, and top ML PhDs recruited to attempt a subset still outperform the models.