Artificial Analysis' AA-Briefcase Benchmark Exposes What AI Models Can't Actually Do
Artificial Analysis launches AA-Briefcase, a new benchmark testing AI agents on real multi-week knowledge work projects with thousands of messy input files
Takeaways
- New benchmark launched: AA-Briefcase tests AI agents on realistic multi-week knowledge work projects with 91 tasks across 4 private scenarios.
- Claude Fable 5 leads with an Elo of 1587, followed by Claude Opus 4.8 (1356) and GLM-5.2 (1266) as the top open-weight model.
- Cost varies 800x: from $0.04/task (DeepSeek V4 Flash) to $31/task (Claude Fable 5), with GLM-5.2 offering the best price/performance.
- Tasks remain very hard: the top model passes all rubric checks on only 3% of tasks; on 31 of 91 tasks, no model scores above 50%.
- Visual inspection predicts quality: top models inspect rendered outputs ~12-21 times per task; low scorers rarely check their own files.
- Open tooling released: benchmark runs on Stirrup, an open-source agent framework, with a public sample scenario on Hugging Face.
Most AI benchmarks ask a model to answer a question, solve a math problem, or write a function. AA-Briefcase asks it to act as a consultant for six weeks, synthesize 25,000 Slack messages and 3,500 emails, build a financial model, and present findings to a board. The gap between those two things is exactly the gap the benchmark is designed to expose.
What the benchmark actually tests
Artificial Analysis built AA-Briefcase around four private, multi-week knowledge work scenarios covering data science, product management, banking operations, and heavy industry strategy. Across those four scenarios there are 91 tasks in total, each requiring a realistic professional deliverable: spreadsheets, PowerPoint decks, PDFs, memos. The tasks build week by week within a scenario, sharing institutional context, so a model working on week 4 needs to understand what happened in weeks 1 through 3.
The input data is deliberately messy. Each task draws on hundreds of files, and the full benchmark contains nearly 2,000 source files, including:
- 25,000+ Slack messages
- 3,500+ emails
- Company documents, meeting transcripts, and large-scale data exports
- Files that contradict each other, simulating real organizational noise
Grading is three-dimensional. Each submission is scored on a binary rubric (did the model find the right numbers, cite the right sources, resolve planted contradictions?), then evaluated pairwise against other models for analytical quality and presentation quality. The combined score is expressed as an Elo rating, the same system used in chess rankings, where a higher number means the model consistently beats lower-ranked models head-to-head.
The leaderboard, and what it costs
Claude Fable 5 achieves the highest AA-Briefcase Elo, which combines rubric pass rate with pairwise analytical quality Elo and presentation quality Elo. It scores 1587 Elo, well ahead of the field. This is followed by Claude Opus 4.8 (max) and GLM-5.2 (max), with GPT-5.5 (xhigh) in fourth.
| Model | Elo | Cost per Task |
|---|---|---|
| Claude Fable 5 | 1587 | ~$31 |
| Claude Opus 4.8 (max) | 1356 | ~$10.40 |
| Claude Opus 4.7 (max) | 1299 | -- |
| GLM-5.2 (max) | 1266 | ~$2.40 |
| GPT-5.5 (xhigh) | 1159 | ~$3.68 |
| MiniMax-M3 | 1116 | -- |
| DeepSeek V4 Flash (max) | 836 | ~$0.04 |
The cost per task on AA-Briefcase varies by more than 800x across models tested. Claude Fable 5 leads the benchmark but costs more than $31 per task on average, compared to ~$0.04 for DeepSeek V4 Flash. The strongest price/performance options are open-weight models such as GLM-5.2 (max) and DeepSeek V4 Pro (max), with GLM-5.2 (max) scoring only ~90 Elo below Claude Opus 4.8 (max) for less than 25% of the cost.
The uncomfortable truth about model performance
Even the top model struggles badly under strict grading. Claude Fable 5 leads overall on rubric pass rate, but satisfies all criteria correctly on only 3% of tasks. On 31 of 91 tasks, no model scores above 50%. That is not a rounding error. It means that on a third of all tasks, every model tested fails to meet even half the ground-truth requirements.
Failure modes shift across model tiers. Less capable models most often fail at task execution, missing relevant input files, submitting unusable deliverables, or producing no deliverable at all. More capable models more often fail to fulfill all task requirements, including those embedded in the original task or hidden across source files.
For each rubric check, the team identifies the minimum set of files a model must read to pass. Across all model capability tiers, pass rates fall as the number of required files increases. High-intelligence models drop from roughly 55% pass rate on prompt-only checks to around 40% when a check requires reading five or more source files. Weaker models fall off much faster.
A surprising signal: visual inspection matters
One of the more unexpected findings is the correlation between a model's use of image inspection and its presentation score. The strongest presentation models inspect their rendered outputs far more often before submitting. Claude Fable 5 and Claude Opus 4.8 (max), the two leading models on presentation Elo, make 21 and 12 visual inspections per task on average respectively, while lower-scoring models inspect much less, with GPT-5.4 Mini at 2 per task and Gemini 3.1 Pro Preview at ~0.1, often submitting files they never visually reviewed.
This suggests that the ability to render a file, look at it, and iterate is a meaningful part of producing polished professional output, not just a nice-to-have. Google's Gemini models show a very different pattern: Gemini 3.5 Flash and Gemma 4 31B are much more compute-heavy, averaging ~60 and ~43 compute calls per task to run scripts or calculations, respectively , but this does not translate into top Elo scores.
How it was built
AA-Briefcase scenarios mirror real-world knowledge work, with tasks developed over months by experts across data science, product management and corporate strategy from companies including Google, McKinsey & Company and Boston Consulting Group. The scenarios are not toy problems adapted from textbooks. They are drawn from the actual professional experience of the people who built them, complete with the ambiguity, competing priorities, and messy context that defines real work.
Every model is run through the same open-source agent harness, Stirrup, which gives each model a consistent environment with tools for file exploration, reading, writing, code execution, and visual inspection. This controls for differences in scaffolding and ensures the leaderboard reflects model capability rather than framework differences. Models can take up to 500 turns per task and submit at any point using a dedicated finish tool.
The four scored scenarios are fully private, including the task instructions, input files, and grading rubrics, to reduce the risk of contamination. A fifth public scenario, AA-Briefcase Lite, has been released on Hugging Face so researchers can see exactly how the evaluation is structured without being able to train or optimize against the real tasks.
What this benchmark changes
The broader benchmark landscape has a saturation problem. Every frontier large language model now scores above 88% on MMLU, the benchmark that defined AI progress for years. GPT-5.3 Codex leads at 93%. At that ceiling, the differences between models are statistical noise. AA-Briefcase is a direct response to that problem, but from a different angle than academic difficulty benchmarks like Humanity's Last Exam. Rather than asking harder questions, it asks longer, messier, more open-ended ones.
The results also challenge a common assumption: that general intelligence scores translate cleanly into agentic performance. AA-Briefcase Elo increases with general intelligence, but the results also highlight different model strengths. MiniMax M3 and GLM-5.2 (max) outperform relative to their Artificial Analysis Intelligence Index score, while Google models such as Gemini 3.5 Flash and Gemini 3.1 Pro Preview underperform on AA-Briefcase relative to their general intelligence ranking. A model that aces GPQA Diamond may still struggle to build a coherent financial model from a folder of contradictory spreadsheets.
For teams evaluating which model to deploy in agentic pipelines for knowledge work, the cost-performance tradeoff is the most actionable takeaway. The best model costs nearly 800 times more per task than the cheapest. GLM-5.2 (max), an open-weight model, sits within 90 Elo points of Claude Opus 4.8 at a fraction of the price. The full AA-Briefcase leaderboard is live and will be updated as new models are released.