AutoLab Reveals Claude Opus Beats Every AI on 40-Hour Engineering Tasks
AutoLab tests 17 frontier models on 36 multi-hour engineering tasks, revealing that persistence beats raw intelligence every time
Takeaways
- New benchmark: AutoLab introduces 36 expert-curated tasks across system optimization, CUDA, model development, and algorithmic puzzles, each running 2-12 hours.
- Claude dominates: claude-opus-4.6 scores 0.68 Avg@3 with 0.93 dominance, beating second-place gemini-3.1-pro (0.50) by an 18-point margin across all four categories.
- Persistence beats intelligence: The dominant predictor of success is not first-attempt quality but the number of edit-benchmark-refine cycles completed; claude averaged 57 agent steps vs. gemini's 12.
- GPT-5.4 and Grok-4-20 underperform: Both rank in the lower half (0.36 and 0.35) due to premature termination, not lack of raw capability.
- Harness choice is huge: The same model's score can shift by 0.43 points depending on the agent harness used, comparable to the gap between top and bottom models.
- Fully open-source: Benchmark, harness, and task artifacts are available at github.com/autolabhq/autolab with a live leaderboard at autolab.moe.
Most AI benchmarks are basically pop quizzes: one question, one answer, done. But real engineering doesn't work that way. Optimizing a CUDA kernel, training a language model, or squeezing latency out of a hash join takes hours of iterative work: profile, hypothesize, edit, rerun, repeat. A new benchmark called AutoLab is designed specifically to test whether frontier models can actually do that, and the results are humbling for most of the field.
The paper evaluates 17 state-of-the-art models across 36 tasks that each run for 2 to 12 wall-clock hours inside a containerized sandbox. The total evaluation consumed 2,544 wall-clock hours and 8.60 billion tokens. The headline finding: the single strongest predictor of success is not how good a model's first attempt is, but whether it keeps iterating, benchmarking, and incorporating feedback throughout the entire budget window.
Why existing benchmarks miss the point
The dominant benchmarks in agentic AI today, things like SWE-bench, GAIA, and LiveCodeBench, evaluate either a single-turn response or a short multi-step trajectory that ends in a final patch. Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons.
The gap matters because the skills required are genuinely different. A model that can write a correct flash-attention implementation in one shot is not the same as a model that can profile it, identify the bottleneck, apply tiled SIMD softmax, re-benchmark, and keep going for 40 minutes until it achieves a 42x speedup. Long-horizon optimization represents a distinct capability that cannot be reduced to agentic coding ability.
What AutoLab actually tests
AutoLab consists of 36 realistic, expert-curated tasks spanning four diverse domains: system optimization, puzzle and challenge, model development, and CUDA kernel optimization. Each task begins with a correct but deliberately suboptimal baseline and challenges agents to improve it within a strict wall-clock budget. Tasks were contributed by senior researchers and engineers drawing from real problems they had personally encountered.

The four categories break down as follows:
- System Optimization (15 tasks): Low-level performance engineering in C, Rust, Go, and Python. Tasks include optimizing a flash-attention kernel, AES-128 encryption throughput, radix sort over 50M integers, and a BVH raytracer.