Skip to main content
Back

Artificial Analysis Reveals Top AI Agents Fail Half of Real Office Tasks

Artificial Analysis launches an independent leaderboard for ServiceNow's enterprise agent benchmark, where even the best model only clears 51% of tasks

  • Artificial Analysis launched EnterpriseOps-Gym-AA, an independent leaderboard for ServiceNow's enterprise agent benchmark across 28 models.
  • Claude Fable 5 leads at 51%, the only model clearly above halfway on 1,117 multi-step enterprise tasks.
  • The benchmark uses 512 tools and 164 database tables across 8 domains, graded by SQL checks on final database state.
  • Strategic planning, not tool use, is the core bottleneck: giving agents human-authored plans boosts scores by 14-35 points.
  • Cost ranges 90x ($0.01 to $0.93 per task); DeepSeek V4 Pro hits 40% at $0.01, within 11 points of the leader.
  • The open-source benchmark is available on GitHub and Hugging Face under Apache 2.0.

Benchmarking AI agents on real enterprise work has always been harder than it sounds. Most evaluations test whether a model can answer questions or write code in isolation. EnterpriseOps-Gym-AA, a new independent leaderboard from Artificial Analysis built in partnership with ServiceNow, tests something much harder: can an agent actually do the job?

A gym built like a real office

EnterpriseOps-Gym was originally developed by ServiceNow Research, Mila, and Université de Montréal. The benchmark drops agents into a containerized sandbox that mimics a live corporate environment. It features 164 database tables and 512 functional tools to mimic real-world search friction. Agents are evaluated across 8 enterprise domains: Calendar, CSM, Drive, Email, HR, ITSM, Teams, and Hybrid, in a fully interactive, containerized environment.

Unlike static QA benchmarks, EnterpriseOps-Gym evaluates agents on final environment state using SQL verifiers, meaning agents are rewarded for achieving the correct outcome, not for following a rigid action sequence. Tasks require long-horizon multi-step reasoning, strict policy compliance, and precise tool invocation under complex data dependencies. The average task takes 9+ steps, and a single bad write can corrupt the environment the same way it would in production.

Artificial Analysis adapted this benchmark to run on their own Stirrup agent harness, running each of the 1,117 oracle-mode tasks 3 times per model across 28 models. The result is an independent, continuously updated leaderboard that sits alongside their broader AI evaluation suite.

The frontier barely clears 50%

The headline number is sobering. Claude Fable 5 (max) leads the board at 51.1%, making it the only model clearly above the halfway mark. Gemini 3.5 Flash (high) follows at 50.1%, and GPT-5.5 (xhigh) sits at 46.6%. These are the most capable models in the world, and they fail roughly half the time on tasks that a competent human employee would handle routinely.

The gap between individual verifier checks and full task success tells the real story. Agents interact through tool calls, and outcomes are verified by SQL-based state checks on the environment database. A task passes only if all verification conditions are met. Claude Fable 5 satisfies 79% of individual verifier checks, but only completes 51% of whole tasks. One missed constraint anywhere in the chain means total failure.

Per-domain performance is wildly uneven

The overall leaderboard ranking hides sharp domain-level swings that matter a lot for practical deployment decisions:

  • GPT-5.5 ranks 3rd overall and is the best single model on Customer Service, but falls to 20th of 28 on Teams.
  • Mistral Medium 3.5 sits 18th overall but ranks 4th on Email, while landing near the bottom on structured HR and ITSM workflows.
  • Claude and Qwen models pull well ahead of GPT-5.5 on Teams specifically.
  • GLM-5.2 (max) from Z AI is the strongest open-weights model at 43%, about 8 points behind the leader, with DeepSeek V4 Pro and Kimi K2.7 both at 40%.

This domain variance is not noise. While a simple two-agent planner-executor architecture delivered modest performance gains, attempts to implement more elaborate decomposition strategies resulted in a surprising regression. In domains with strong sequential dependencies, such as Customer Service Management and Human Resources, breaking a task into sub-problems for different agents consistently disrupted the contextual continuity required for success.

The real bottleneck is planning, not tool use

The original paper surfaces a finding that should reshape how teams think about enterprise agents. Providing oracle human plans improves performance by 14-35 percentage points, pinpointing strategic reasoning as the primary bottleneck. Models are not failing because they cannot call the right API. They fail because they cannot figure out the right sequence of steps when no one has mapped it out for them.

Tool selection and execution remained stable even under adversarial conditions. Models don't fail at the point of action. They fail at knowing what to do across a constrained, multistep workflow when no one has mapped it out for them.

There is a second, quieter failure mode: agents frequently fail to refuse infeasible tasks, with the best model achieving only 53.9% clean abstention, leading to unintended and potentially harmful side effects. In a real enterprise, an agent that confidently executes a policy-violating request is worse than one that simply fails.

Efficiency and cost are all over the map

Cost per task ranges roughly 90x across models, from about $0.01 for DeepSeek V4 Pro to $0.93 for Claude Fable 5. But higher spend does not reliably buy a higher score. DeepSeek V4 Pro reaches 40% at $0.01 per task, within 11 points of the leader at a fraction of the cost. Claude Opus 4.8, the second most expensive model at $0.79 per task, reaches only 44%.

Turn efficiency also varies dramatically and in unexpected ways:

  • Claude Fable 5 averages 6.5 turns per task to reach its 51% score.
  • Gemini 3.5 Flash uses 13.5 turns, making more attempts before giving up.
  • Grok 4.5 uses 17.4 turns but underperforms because it gets stuck in loops, repeatedly calling the same tools without making progress.

How to run it yourself

The benchmark comprises 1,150 expert-curated tasks spanning 8 enterprise domains, each running against live containerized MCP servers backed by realistic, fully synthetic databases. The full codebase is open source under Apache 2.0. Setup requires Python 3.11+ and Docker. Each domain runs as a separate MCP server container, and you configure your model via a simple JSON file:

git clone https://github.com/ServiceNow/EnterpriseOps-Gym.git
cd EnterpriseOps-Gym
uv sync --extra openai  # or anthropic, google, deepseek, all
# Pull and start the MCP server for each domain
docker pull shivakrishnareddyma225/enterpriseops-gym-mcp-teams:latest
docker run -d -p 8002:8002 shivakrishnareddyma225/enterpriseops-gym-mcp-teams:latest
# Run evaluation
python evaluate.py \
    --hf_dataset ServiceNow-AI/EnterpriseOps-Gym \
    --domain teams --mode oracle \
    --llm_config conf/llm/gpt-4.1-mini.json \
    --output_folder results/react/gpt-4.1-mini/teams/oracle \
    --orchestrator react --concurrency 4

The dataset is also available directly on Hugging Face if you want to inspect tasks without running the full environment. The independent leaderboard on Artificial Analysis is live now and free to browse.

Why this benchmark matters now

Deployment of LLMs as reliable AI workers in enterprise is stalled by benchmarks that fail to capture the intricacies of professional environments, specifically the need for long-horizon planning amidst persistent state changes and strict access protocols. EnterpriseOps-Gym is a direct response to that gap.

NOWAI-Bench, an open benchmarking suite for enterprise AI agents integrated with the NVIDIA NeMo Gym library, includes EnterpriseOps-Gym as one of the industry's most challenging enterprise agent benchmarks. The benchmark is already attracting serious institutional attention, which means the leaderboard will only get more competitive from here.

The practical takeaway for teams building on top of frontier models: a model's headline score on coding or reasoning benchmarks tells you almost nothing about how it will perform when asked to update an HR record, resolve an IT ticket, or coordinate a cross-domain workflow without a human in the loop. The 50% ceiling at the frontier is not a minor gap to close. It is a signal that the planning layer, not the model itself, is where the real engineering work needs to happen.