Artificial Analysis's AA-AnalystAgent Benchmark Reveals Claude Opus 5 Beats GPT-5.5 on Reliability
Artificial Analysis launches AA-AnalystAgent, a new benchmark that tests AI agents on real spreadsheets — and even the best model only passes 54% of tasks every time.
- New benchmark: AA-AnalystAgent tests AI agents on 80 real-world quantitative analysis tasks across 14 domains using actual spreadsheets and documents.
- Headline metric is pass^5: Models must answer each task correctly on all 5 independent attempts — not just once — to count it as solved.
- Top scores are low: Claude Opus 5 leads at 54%, GPT-5.5 at 50%, Claude Fable 5 at 49%; even the best model fails nearly half the time.
- Reliability beats raw accuracy: GPT-5.5 has the highest single-attempt score (66%), but Opus 5 wins on pass^5 by being more consistent across runs.
- Cost is unpredictable: Two models scoring 20% cost $1.34 and $0.05 per task respectively; expensive models don't reliably outperform cheaper ones.
- Open-source harness: Stirrup, the agent framework used to run all tasks, is publicly available on GitHub under MIT license.
Most AI benchmarks test whether a model can get the right answer once. But if you're deploying an agent to do analyst work, "sometimes correct" is not good enough. That's the core insight behind AA-AnalystAgent, a new agentic benchmark from Artificial Analysis that tests models on real-world quantitative analysis tasks , and requires them to get each one right on all five independent attempts.
What it actually tests
The benchmark covers 80 questions across 14 business and scientific domains. These aren't toy problems. The source material includes actual government expenditure reports, commodity trade statistics, hydrology datasets, energy cost models, and financial valuation spreadsheets. Tasks span five workflow types that mirror real analyst work:
- Source lookup and diagnosis
- Filter and total
- Ratios, trends, and sensitivities
- P&L modeling
- Cash flow, balance sheet, and valuation modeling
The question set is kept private to reduce contamination risk. Two example tasks drawn from California Medicaid expenditure reports are publicly shown on the methodology page, complete with full prompts and source material.
The metric that changes everything
When businesses and researchers hand analyst work to an AI agent, they need to be confident it will produce the correct answer every time. That's why AA-AnalystAgent runs every task five times and reports pass^5 as its headline metric. A task only counts as solved if the model answers correctly on all five independent attempts.