Skip to main content
Back

Artificial Analysis' AA-Briefcase Reveals Top AI Agents Fail 97% of Real Work Tasks

AA-Briefcase reveals frontier agents take 10–25 minutes per task on realistic knowledge work, exposing a new speed-quality tradeoff that existing benchmarks miss entirely

  • AA-Briefcase launched: Artificial Analysis released a new proprietary agentic benchmark testing models on realistic multi-week knowledge work projects built by Google, McKinsey, and BCG experts.
  • Frontier models take 10–25 min per task: Claude Opus 4.8 averages ~23 min; GPT-5.5 (xhigh) hits top-5 Elo in ~11 min, making it the best speed-quality tradeoff.
  • Even the best model fails most tasks: Claude Fable 5 leads at 1587 Elo but passes all rubric checks on only 3% of tasks; 31 of 91 tasks have no model above 50% pass rate.
  • 800x cost spread: Claude Fable 5 costs $31+/task vs. ~$0.04 for DeepSeek V4 Flash; GLM-5.2 is the best open-weights value, scoring ~90 Elo below Opus 4.8 at under 25% of the cost.
  • Tool execution is not the bottleneck: Only ~12% of task time is tool calls; output verbosity and turn count drive the rest.
  • AA-Briefcase Lite is public: A sample due diligence scenario is available on Hugging Face; the benchmark runs on the open-source Stirrup agent framework.

Most AI benchmarks measure what a model knows. AA-Briefcase measures what a model can actually do , and the results are a reality check for anyone deploying agents on real knowledge work. Artificial Analysis just released this proprietary benchmark, and it is unlike anything in the current evaluation landscape.

Not your typical benchmark

The core idea is simple but demanding: put an AI agent inside a realistic multi-week professional project and see if it can produce actual deliverables. AA-Briefcase is a private evaluation for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos, across four multi-week knowledge work projects comprising thousands of input files and 91 tasks in total.

The four project scenarios span:

  • Data Science , transaction data cleaning, forecast modeling, schema design
  • Product Management , competitive teardowns, PRD writing, go-to-market planning
  • Banking Operations , branch network transformation, financial modeling, mortgage journey mapping
  • Heavy Industry Strategy , commodity supply-demand models, M&A valuation benchmarking

Tasks build week by week, draw on shared institutional context, and require deliverables such as financial models, board presentations, and design mock-ups. The source material is fragmented, messy, and often contains realistic contradictions, testing whether models can navigate the ambiguity of real-world knowledge work. In total, the benchmark contains nearly 2,000 source files, including 25,000+ Slack messages and 3,500+ emails.

How it scores

Scoring is a three-part composite. Each task is graded on:

  1. Rubric pass rate , binary pass/fail checks for ground-truth correctness (did the model find the right files, reach the right conclusions?)
  2. Analytical quality Elo , pairwise comparison: which model's output is more rigorous and well-supported?
  3. Presentation Elo , pairwise comparison: which output is more professionally presented?

All grading and pairwise comparisons are decided by one of a panel of three judges , Claude Opus 4.8 (max), GPT-5.5 (xhigh), and Gemini 3.1 Pro Preview , rather than a single judge, reducing bias toward submissions from the same model or model family. Each rubric verdict and pairwise comparison is decided by one judge sampled from the panel, with sampling balanced across checks and matches.

The benchmark runs on Stirrup, Artificial Analysis's open-source agent framework. Stirrup is a lightweight framework for building agents that differs from others by getting out of the way and letting the model choose its own approach to completing tasks, similar to Claude Code. Many frameworks impose rigid workflows that can degrade results.

The leaderboard , and the gap at the top

Claude Fable 5 achieves the highest AA-Briefcase Elo, combining rubric pass rate with pairwise analytical quality and presentation quality Elo. This is followed by Claude Opus 4.8 (max) and GLM-5.2 (max), with GPT-5.5 (xhigh) in fourth. But the headline number masks a brutal truth about how hard the benchmark actually is.

Despite leading the rankings, Claude Fable 5 fully satisfies all evaluation criteria on only 3% of tasks. Across 31 of the 91 tasks, no AI model managed to achieve even a 50% pass rate. The results suggest that current frontier AI models remain far from reliably handling complex, long-duration knowledge work.

Failure modes also shift depending on model capability tier:

  • Weaker models tend to fail at basic execution tasks such as finding relevant files, understanding context, or producing usable outputs.
  • More capable models more often fail to fulfill all task requirements, including those hidden across source files , incorrect or unfinished analysis and formatting errors remain common across all tiers.

The speed-quality tradeoff nobody was measuring

The most actionable insight from AA-Briefcase is not who ranks first , it is how long each model takes to get there. This is the metric that matters most for production deployments.

Agentic knowledge work can take frontier models over 20 minutes per task, as measured in AA-Briefcase. Here is how the top models stack up:

ModelAA-Briefcase EloAvg. Time per TaskCost per Task
Claude Fable 51587 (not available)~28.5 min (est.)~$31+
Claude Opus 4.8 (max)1356~23–24 minHigh
GLM-5.2 (max)1261~16.3 min<25% of Opus 4.8
GPT-5.5 (xhigh)1159~11 minMid
DeepSeek V4 Flash (max)LowFast~$0.04

GPT-5.5 in its various reasoning configurations sits on the Pareto frontier , the set of models where you cannot improve on one dimension without sacrificing another. GPT-5.5 (xhigh) in particular stands out as one of the most efficient top-performing models, using around half the time per task of Opus 4.8 at 11 minutes, while ranking top 5 on overall AA-Briefcase Elo.

GLM-5.2 also sits on the Pareto frontier, scoring 1261 , ahead of GPT-5.5 (xhigh) but also taking more time per task at 16.3 minutes. It is also the top-performing open-weights model on AA-Briefcase, with MiniMax-M3 the next best at 1113.

Crucially, time spent on tool calls and execution accounts for only ~12% of the total time, with the remaining amount explained by output verbosity, turn usage, and inference speed. In other words, the bottleneck is not tool execution , it is how much the model writes and how many turns it takes.

Visual inspection is a secret weapon

One of the more surprising findings: models that visually inspect their own rendered outputs before submitting score dramatically higher on presentation quality. Claude Opus 4.8 is the highest-scoring available model, but it is also one of the slowest, taking ~23 minutes per task on average. Both top Anthropic models make heavy use of the view_image tool , Claude Fable 5 averages 21 visual inspections per task, Opus 4.8 averages 12. Lower-scoring models like GPT-5.4 Mini average just 2, and Gemini 3.1 Pro Preview averages ~0.1, often submitting files they never looked at.

Cost: an 800x spread

Cost per task varies by ~800x across models tested: Claude Fable 5 leads the benchmark but costs more than $31 per task on average, compared to ~$0.04 for DeepSeek V4 Flash (max). None of the cheapest models reach frontier performance, but the open-weight middle ground is compelling. The strongest price/performance options are open-weights models such as GLM-5.2 (max) and DeepSeek V4 Pro (max). GLM-5.2 scores only ~90 Elo below Claude Opus 4.8 for less than 25% of the cost.

Why this benchmark matters now

The existing agent benchmark landscape , SWE-bench for coding, GAIA for general assistant tasks, WebArena for browser navigation , was built around short, discrete tasks with clear right-or-wrong answers. AA-Briefcase simulates the kind of complex information environments faced by analysts, consultants, researchers, and corporate employees. That is a fundamentally different problem.

The benchmark also surfaces a gap that general intelligence scores miss entirely. Google's Gemini models, for example, underperform on AA-Briefcase relative to their general intelligence ranking. High general intelligence does not automatically translate to strong knowledge work performance when the task requires navigating thousands of messy, contradictory source files and producing a polished deliverable.

The four private scenarios remain held out to reduce contamination risk. A public fifth scenario , a due diligence project , is available on Hugging Face as AA-Briefcase Lite for anyone who wants to explore the benchmark structure. The full leaderboard is updated as new models are released.

The practical takeaway for teams building agentic pipelines: raw capability scores are only half the story. A model that takes 23 minutes per task at $31 a run is a very different infrastructure decision than one that takes 11 minutes at a fraction of the cost , even if the quality gap is modest. AA-Briefcase is the first benchmark that forces you to confront that tradeoff with real data.