
Benchmarks that test whether a model can answer trivia or solve math problems are increasingly beside the point. What practitioners actually need to know is: can this model sit down, read through a thousand messy files, and produce a polished PowerPoint or a defensible financial model? That is exactly what AA-Briefcase is designed to test, and GPT-5.6 Sol just posted a result worth paying attention to.
What AA-Briefcase actually measures
AA-Briefcase is a private evaluation developed by Artificial Analysis for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos. It evaluates models across four multi-week knowledge work projects, comprising thousands of input files and 91 tasks in total, across fields such as data science, product management, and corporate strategy.
The four private scenarios are:
- Data Science -- quantitative work on imperfect datasets turned into business recommendations
- Product Management -- competitive teardowns, PRD writing, and go-to-market planning
- Banking Operations -- branch network transformation, financial modeling, and mortgage journey mapping
- Heavy Industry Strategy -- 10-year commodity models, M&A comps, and asset-level operating models
Unlike traditional benchmarks, it requires models to work through thousands of fragmented documents, including Slack conversations, emails, meeting transcripts, spreadsheets, and large-scale data exports. The benchmark simulates the kind of complex information environments faced by analysts, consultants, researchers, and corporate employees.
The sources are fragmented, contradictory, and full of realistic ambiguity. A memo from week one might directly conflict with an email from week three. The relevant signal is buried under irrelevant noise.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves

