Artificial Analysis's AA-AnalystAgent Benchmark Reveals Claude Opus 5 Beats GPT-5.5 on Reliability

Most AI benchmarks test whether a model can get the right answer once. But if you're deploying an agent to do analyst work, "sometimes correct" is not good enough. That's the core insight behind AA-AnalystAgent, a new agentic benchmark from Artificial Analysis that tests models on real-world quantitative analysis tasks , and requires them to get each one right on all five independent attempts.

What it actually tests

The benchmark covers 80 questions across 14 business and scientific domains. These aren't toy problems. The source material includes actual government expenditure reports, commodity trade statistics, hydrology datasets, energy cost models, and financial valuation spreadsheets. Tasks span five workflow types that mirror real analyst work:

• Source lookup and diagnosis
• Filter and total
• Ratios, trends, and sensitivities
• P&L modeling
• Cash flow, balance sheet, and valuation modeling

The question set is kept private to reduce contamination risk. Two example tasks drawn from California Medicaid expenditure reports are publicly shown on the methodology page, complete with full prompts and source material.

The metric that changes everything

When businesses and researchers hand analyst work to an AI agent, they need to be confident it will produce the correct answer every time. That's why AA-AnalystAgent runs every task five times and reports pass^5 as its headline metric. A task only counts as solved if the model answers correctly on all five independent attempts.

This distinction , pass^5 versus the more familiar pass@1 , reshapes the leaderboard dramatically. Gemini 3.1 Pro Preview solves 81% of tasks at least once but only 41% on all five, and finishes ninth. The benchmark's argument is blunt: an analyst agent is only useful if its answers hold up without re-checking , an answer that is right at random still has to be verified, which is the work the agent was meant to remove.

Who's on top

Claude Opus 5 leads at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49%. Anthropic holds three of the top five places, and the top three are separated by four net tasks out of 80.

Reliability separates the top of the leaderboard more than raw capability: GPT-5.5 has the highest pass@1, but Opus 5 leads on pass^5 because it repeats what it gets right. GPT-5.5 gets a higher average score on any given attempt (66% pass@1), but Opus 5 is more consistent , it re-executes the same correct workflow reliably across all five runs.

On the open-weights side: Kimi K3 is the top open weights model at 39%, 15 points behind the closed frontier. The next open weights model is DeepSeek V4 Flash at 25%, so the spread inside the open weights field is currently wider than the gap from its leader to the frontier.

How models fail

1,567 failing attempts were classified across ten leading models into seven failure modes. The most widespread is anchoring on a wrong early hypothesis, present in 57% of failures. Models commit to a source or interpretation early and defend it for the rest of the trajectory.

The failure signatures vary sharply by model. Gemini 3.1 Pro Preview takes sources at their word but fumbles the execution, posting above-median rates of modeling, scaling and aggregation errors (51% of its failures). Grok 4.5 is the reverse , it understands the field's language, then substitutes its own assumptions for what the documents say, overriding evidence on 54% of its failures against a 43% median.

The cost picture is messy

The price of a given score varies enormously. Claude Sonnet 4.6 and MiMo-V2.5-Pro both score 20%, at $1.34 and $0.05 per task. Claude Opus 4.7 is the most expensive model to run at $1.98 per task and places eighth, while GPT-5.5 runs it for $1.15 and places second. The Flash tier is not automatically the cheaper option either, with Gemini 3.5 Flash at $1.79 against Gemini 3.1 Pro Preview at $0.80. Cost-efficiency is far from predictable , you can't just assume a cheaper model tier means a cheaper run.

The open-source harness behind it

All tasks are run through Stirrup, Artificial Analysis's open-source agent framework. Stirrup differs from other agent frameworks by getting out of the way and letting the model choose its own approach to completing tasks, similar to Claude Code. Many frameworks impose rigid workflows that can degrade results. Best practices around context management and foundational tools are built in.

You can install it and run your own agents with a few lines:

pip install stirrup
# Or with all optional components
pip install 'stirrup[all]'

Stirrup supports OpenAI-compatible APIs, LiteLLM (for Anthropic, Google, etc.), Docker and E2B sandboxes for code execution, MCP server connections, and multimodal inputs. It's already at 513 stars on GitHub with 57 forks.

What this benchmark is actually for

AA-AnalystAgent is most useful if you're evaluating whether to deploy an agent on analyst-style workflows , financial modeling, regulatory reporting, data reconciliation , where the cost of a wrong answer is high and re-checking defeats the purpose. The benchmark surfaces two things that standard leaderboards miss: whether a model is consistent enough to trust without verification, and whether the cost of running it is proportional to the quality you get.

The leaderboard is live and standalone , it's not rolled into the Artificial Analysis Intelligence Index. New models will be added as they're released. The full methodology, including grading prompts and harness configuration, is documented on the methodology page. Even the best model today sits at 54% on pass^5 , which means for any workflow where you need consistent correctness, human review is still in the loop.