Most AI benchmarks tell you how well a model solves math problems or writes code. AutomationBench-AA, a new independent leaderboard from Artificial Analysis built in partnership with Zapier, asks a harder question: can an AI agent actually do the work a business runs on? The answer, across every model tested, is a firm "not yet" -- though the gap between leaders is already significant and the cost differences are striking.
The benchmark the enterprise was missing
Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real business workflows demand all three: a single task may span a CRM, inbox, calendar, and messaging platform -- requiring the agent to find the right endpoints, follow a policy document, and write correct data to each system. That gap is exactly what AutomationBench was designed to fill.
AutomationBench tests AI agents on end-to-end workflow execution using 47 real tools across six business functions: Sales, Marketing, Operations, Support, Finance, and HR. It's built on real patterns from 2B+ monthly tasks across 3.7M companies. Zapier initially built it for internal use to decide which models to deploy on their own platform, and then open-sourced it.
How it works
Agents receive a natural language business task and must autonomously discover available REST API endpoints across dozens of integrated applications, make sequential and interdependent calls, apply layered business rules, and navigate environments seeded with distractors, decoy records, and adversarial inputs. Think stale CRM rows, near-duplicate contact names, and policies buried inside a spreadsheet linked from an email thread.
Scoring is deliberately unforgiving. Grading is programmatic and end-state only: whether the correct data ended up in the right systems. The agent's reply isn't scored -- the data is. There's no LLM-as-judge, no partial credit for good reasoning that produces the wrong outcome.
The benchmark also introduces a two-part scoring system that sets AutomationBench-AA apart from Zapier's own hosted leaderboard:
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves