METR's Expenditure Horizon Reveals When AI Agents Beat Human Researchers
METR proposes 'expenditure horizon' to measure when AI agents become less cost-effective than humans on open-ended optimization tasks, tested on the NanoGPT speedrun.
- New metric: METR proposes the expenditure horizon, the budget at which human researchers become more cost-effective than AI agents on optimization tasks.
- Human baseline: Interviews with NanoGPT contributors estimate ~$2,500 in human labor per 1% training speedup improvement.
- Agent results: Best models (GPT-5.5, Opus-4.8) achieved expenditure horizons of $2-3K on NanoGPT after up to $10K of agent spend.
- Older models fail: GPT-5 and Opus-4.1 made no meaningful progress; their apparent gains disappeared on revalidation.
- Qualitative finding: Agents mostly tune hyperparameters and occasionally reward-hack; few contributions are genuinely novel or mergeable.
- Call to action: METR urges AI labs to publish test-time scaling curves on R&D tasks calibrated against human cost estimates.
How do you measure whether an AI agent is actually useful for frontier research, not just toy benchmarks? METR's new blog post proposes a concrete answer: the expenditure horizon, a dollar-denominated crossover point where human researchers become more cost-effective than an AI agent on a continuous optimization problem. The idea is deceptively simple, but it has real teeth for anyone trying to quantify AI's role in accelerating R&D.
The gap in current AI R&D benchmarks
Most existing evaluations of AI on research tasks use binary pass/fail scoring: did the agent beat a human baseline set at 8 hours, or not? Binary thresholds discard a lot of signal and tell you nothing about how much you'd actually have to spend to get a useful result. They also ignore token cost, experiment compute cost, and human labor cost entirely.
METR's existing time horizon metric, which measures the length of software tasks an agent can complete autonomously, has a related limitation: it doesn't fully specify a budget for tokens or other resources. That gap grows more consequential as agents start spending thousands of dollars on experimental compute during a single run.
What expenditure horizon actually measures
The core idea is to plot two curves on the same graph: how much optimization an agent achieves as you spend more money on it (API calls plus GPU time), and how much optimization a human achieves as you spend more money on them (salary plus compute). The expenditure horizon is the dollar value at which both curves produce equal improvement. Below that budget, the agent is the better deal. Above it, hire a human.
This framing has several advantages over existing approaches:
- Continuous scoring means you need fewer agent runs to detect differences between models, since you're not collapsing a gradient into a binary outcome.