Benchmarks that test whether a model can answer trivia or solve math problems are increasingly beside the point. What practitioners actually need to know is: can this model sit down, read through a thousand messy files, and produce a polished PowerPoint or a defensible financial model? That is exactly what AA-Briefcase is designed to test, and GPT-5.6 Sol just posted a result worth paying attention to.

What AA-Briefcase actually measures

AA-Briefcase is a private evaluation developed by Artificial Analysis for frontier agentic capability in long-horizon knowledge work, testing agents on realistic business workflows that require deliverables such as spreadsheets, presentations, and memos. It evaluates models across four multi-week knowledge work projects, comprising thousands of input files and 91 tasks in total, across fields such as data science, product management, and corporate strategy.

The four private scenarios are:

  • Data Science -- quantitative work on imperfect datasets turned into business recommendations
  • Product Management -- competitive teardowns, PRD writing, and go-to-market planning
  • Banking Operations -- branch network transformation, financial modeling, and mortgage journey mapping
  • Heavy Industry Strategy -- 10-year commodity models, M&A comps, and asset-level operating models

Unlike traditional benchmarks, it requires models to work through thousands of fragmented documents, including Slack conversations, emails, meeting transcripts, spreadsheets, and large-scale data exports. The benchmark simulates the kind of complex information environments faced by analysts, consultants, researchers, and corporate employees.

The sources are fragmented, contradictory, and full of realistic ambiguity. A memo from week one might directly conflict with an email from week three. The relevant signal is buried under irrelevant noise.

Alpha Signal

Don't miss what's next in AI

Join 300,000+ engineers and researchers who get the signal, not the noise.

  • Full access to in-depth AI research breakdowns
  • Be the first to know what's trending before it hits mainstream
  • Daily curated papers, repos, and industry moves