--- title: "IBM's ITBench-AA Shows Every Frontier AI Agent Failing Over Half of IT Tasks" subtitle: "IBM and Artificial Analysis launch ITBench-AA, a Kubernetes SRE benchmark where every frontier model fails more than half the time — and more turns make things worse." url: "https://alphasignal.ai/news/ibm-s-itbench-aa-shows-every-frontier-ai-agent-failing-over-half-of-it-tasks" author: "AlphaSignal Newsroom" author_url: "https://alphasignal.ai/editorial-team" company: "Artificial Analysis" company_url: "https://alphasignal.ai/company/artificialanlys" published: "2026-05-27T18:08:43.000Z" updated: "2026-06-08T19:33:17.269Z" type: "News" topics: - "Benchmarks" - "Agents" subtopics: - "Agent Frameworks" - "Deep Research" upvotes: 654 read_time: "6 min" source_url: "https://huggingface.co/datasets/ArtificialAnalysis/ITBench-AA/tree/main/sre" comments: 0 --- # IBM's ITBench-AA Shows Every Frontier AI Agent Failing Over Half of IT Tasks > IBM and Artificial Analysis launch ITBench-AA, a Kubernetes SRE benchmark where every frontier model fails more than half the time — and more turns make things worse. Every major AI lab now ships some version of an autonomous SRE agent. Every major observability vendor ships an AI SRE agent now , Datadog, AWS, PagerDuty, incident.io , all of them report 40-70% MTTR reduction in their own testing. The problem is that those numbers come from the vendors selling you the agent. Now there is a controlled, independent answer: the first controlled, independent agentic enterprise IT benchmark is live, with Artificial Analysis and IBM testing 59 Kubernetes incident scenarios against every major frontier model , and all of them failed more than half. ## What ITBench-AA actually is Artificial Analysis and IBM Software Innovation Lab are launching [ITBench-AA](https://artificialanalysis.ai/evaluations/itbench-aa), the first in a new series of benchmarks evaluating models on agentic enterprise IT tasks, starting with Site Reliability Engineering tasks where frontier models score below 50%. The underlying ITBench dataset has been developed by IBM, leveraging deep expertise in enterprise IT operations. Artificial Analysis has worked closely with IBM over the last 6 months to develop an implementation of the dataset for frontier AI evaluation, beginning with Site Reliability Engineering (SRE) and expanding to Financial Operations (FinOps) and Chief Information Security Officer (CISO) tasks over time. SRE (Site Reliability Engineering) is the discipline responsible for keeping production systems alive. When something breaks at 2 AM, an SRE investigates alerts, reads logs, traces service dependencies, and identifies the root cause. ITBench-AA asks AI agents to do exactly that. In each scenario, the agent is given an offline snapshot of a Kubernetes incident containing alerts, events, traces, metrics, and application topology, and must produce a structured JSON diagnosis identifying the root cause entities responsible for the failure. The benchmark evaluates 59 Kubernetes incident tasks: 40 from IBM's public release and 19 private tasks shared by the ITBench team, with 3 repeats per task. Faults span realistic failure modes: resource quota exhaustion, rollout failures, connection pool exhaustion, and network partitions. ## The harness: Stirrup All models are evaluated through [Stirrup](https://stirrup.artificialanalysis.ai/), Artificial Analysis's open-source agent harness. All models are run using the open-source agentic harness Stirrup, with a 100-turn cap per task. Keeping the harness constant across all models is what makes this an apples-to-apples comparison , the only variable is the model itself. Stirrup is also available for your own projects. A minimal setup looks like this: ``` pip install stirrup from stirrup import Agent from stirrup.clients.chat_completions_client import ChatCompletionsClient import asyncio async def main(): client = ChatCompletionsClient( base_url="https://openrouter.ai/api/v1", model="anthropic/claude-opus-4-5", ) agent = Agent(client=client, name="sre_agent", max_turns=100) async with agent.session(output_dir="output/") as session: await session.run("Investigate the incident snapshot in /data and identify root causes.") asyncio.run(main()) ``` ## The scoring is deliberately unforgiving The metric used is **average precision at full recall**. This means: if the agent misses even one true root cause, it scores zero for that run. If it finds all of them, it scores based on precision , the fraction of submitted entities that are actually correct. Submitting extra entities beyond the real root cause counts against you. Models that submit additional contributing entities beyond the true root cause get penalized: identifying the correct root cause but adding upstream mechanisms (like a chaos-mesh controller) or co-occurring symptoms counts as a false positive under recall-gated precision. This is intentional. In a real incident, a diagnosis that points to five possible causes when only one is responsible wastes the on-call engineer's time and erodes trust in the system. ## No frontier model cracks 50% Claude Opus 4.7 (Adaptive Reasoning, Max Effort) scores the highest on ITBench-AA with 46.7%, followed by GPT-5.5 (xhigh) with 45.8%, and Qwen3.7 Max with 42.5%. The full leaderboard breakdown: - **Claude Opus 4.7** (Adaptive Reasoning, Max Effort): 47% , highest score, $5.38/task - **GPT-5.5** (xhigh): 46% , 31 avg turns/task, $X/task - **Qwen3.7 Max**: 42% - **GLM-5.1** (Reasoning): 40% , leads open-weight models, $1.23/task - **Gemini 3.5 Flash** (high): 40% , effectively tied with GLM-5.1, $1.70/task - **DeepSeek V4 Pro** (Reasoning, Max Effort): 38% - **Gemma 4 31B** (Reasoning): 37% , $0.14/task - **Gemini 3.1 Pro Preview**: 30% , 83 avg turns/task, $2.23/task All frontier models score below 50%, making ITBench-AA SRE one of the least saturated agentic benchmarks in Artificial Analysis's suite. For context, frontier models score considerably higher on Terminal-Bench. This is not a benchmark that has been quietly solved. ## The turn-count paradox The most counterintuitive finding in the results: **more investigation makes things worse**. Turn counts vary nearly 3x and longer trajectories do not translate to higher accuracy. GPT-5.5 (xhigh) averages 31 turns per task at 46%, while Gemini 3.1 Pro Preview averages 83 turns at 30%. Models that over-investigate tend to surface upstream fault-injection mechanisms or co-occurring symptoms as false positives. In a Kubernetes incident snapshot, the signal is dense: alerts fire across multiple layers simultaneously, events propagate through dependent services, traces catch co-occurring anomalies that aren't the root cause. An agent that keeps investigating past the point of commitment starts surfacing upstream fault-injection artifacts and correlated symptoms instead of locking an answer. The skill being tested is not thoroughness , it is knowing when to stop. ## Cost vs. accuracy: the real decision surface The benchmark surfaces a cost-performance tradeoff that is directly actionable. Cost per task ranges from $0.14 to $5.38, making cost and turn efficiency as decisive as raw score for vendor selection. | Model | Score | Avg Turns | Cost/Task | | --- | --- | --- | --- | | Claude Opus 4.7 (Adaptive, Max) | 47% | -- | $5.38 | | GPT-5.5 (xhigh) | 46% | 31 | -- | | GLM-5.1 (Reasoning) | 40% | -- | $1.23 | | Gemini 3.5 Flash (high) | 40% | -- | $1.70 | | Gemma 4 31B (Reasoning) | 37% | -- | $0.14 | | Gemini 3.1 Pro Preview | 30% | 83 | $2.23 | Open-weight models offer competitive cost-performance tradeoffs, with Gemma 4 31B scoring 37% at only $0.14 per task versus proprietary models costing up to $5.38 per task. The 38x cost gap between the cheapest and most expensive model for a 10-point accuracy difference is not a rounding error , it is an architectural decision for any team running agents at scale. ## What this means for how you build A single-model architecture covering the full enterprise IT agent perimeter is no longer defensible on these figures. Routing by incident criticality and type becomes a first-class architectural decision, not an optimization to revisit later. Practically, the benchmark suggests a tiered approach: - **High-criticality, low-frequency incidents** (network partitions, complex rollout failures): Claude Opus 4.7 or GPT-5.5 despite the cost - **High-volume, recurring diagnostics** (quota monitoring, standard application alerts): Gemma 4 31B or GLM-5.1, with documented cost-performance ratios - **Turn count matters as much as score**: an agent averaging 83 turns introduces latency that is structurally incompatible with real-time alerting pipelines The benchmark is live and fully open. IBM built the framework on CrewAI, deployed to Kubernetes, and published it at [github.com/itbench-hub/ITBench](https://github.com/itbench-hub/ITBench). Any team can submit to the Artificial Analysis leaderboard. This isn't a press-release number from a vendor , it's a submittable, reproducible benchmark with public methodology. The [dataset is on HuggingFace](https://huggingface.co/datasets/ArtificialAnalysis/ITBench-AA/tree/main/sre) and the leaderboard tracks score, cost-per-task, and turn count side by side. The FinOps and CISO domains are coming next, but for now the SRE results alone are enough to recalibrate expectations about where AI agents actually stand on enterprise IT work. ## Sources - [X post by @artificialanlys](https://x.com/artificialanlys/status/2059698327235805258) - [ITBench-AA Benchmark Leaderboard | Artificial Analysis](https://artificialanalysis.ai/evaluations/itbench-aa) - [ITBench-AA: Frontier Models Score Below 50% — Hugging Face Blog](https://huggingface.co/blog/ibm-research/itbench-aa) - [Stirrup — Artificial Analysis Open-Source Agent Harness](https://stirrup.artificialanalysis.ai/) - [ITBench GitHub — itbench-hub/ITBench](https://github.com/itbench-hub/ITBench) - [Every Frontier Model Fails the Majority of SRE Incidents — AI Founders](https://aifounders.cz/en/every-frontier-model-fails-the-majority-of-sre-incidents-itbench-aa-finally-has-numbers/)