Benchmarking AI agents on real enterprise work has always been harder than it sounds. Most evaluations test whether a model can answer questions or write code in isolation. EnterpriseOps-Gym-AA, a new independent leaderboard from Artificial Analysis built in partnership with ServiceNow, tests something much harder: can an agent actually do the job?
A gym built like a real office
EnterpriseOps-Gym was originally developed by ServiceNow Research, Mila, and Université de Montréal. The benchmark drops agents into a containerized sandbox that mimics a live corporate environment. It features 164 database tables and 512 functional tools to mimic real-world search friction. Agents are evaluated across 8 enterprise domains: Calendar, CSM, Drive, Email, HR, ITSM, Teams, and Hybrid, in a fully interactive, containerized environment.
Unlike static QA benchmarks, EnterpriseOps-Gym evaluates agents on final environment state using SQL verifiers, meaning agents are rewarded for achieving the correct outcome, not for following a rigid action sequence. Tasks require long-horizon multi-step reasoning, strict policy compliance, and precise tool invocation under complex data dependencies. The average task takes 9+ steps, and a single bad write can corrupt the environment the same way it would in production.
Artificial Analysis adapted this benchmark to run on their own Stirrup agent harness, running each of the 1,117 oracle-mode tasks 3 times per model across 28 models. The result is an independent, continuously updated leaderboard that sits alongside their broader AI evaluation suite.
The frontier barely clears 50%
The headline number is sobering. Claude Fable 5 (max) leads the board at 51.1%, making it the only model clearly above the halfway mark. Gemini 3.5 Flash (high) follows at 50.1%, and GPT-5.5 (xhigh) sits at 46.6%. These are the most capable models in the world, and they fail roughly half the time on tasks that a competent human employee would handle routinely.
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves