Every hardware benchmark you've ever used was built for a world that no longer exists. Synthetic prompts, fixed input lengths, no KV cache reuse, no speculative decoding , the numbers look clean, but they describe a deployment style nobody actually ships. Meanwhile, the real workload of 2026 is a coding agent that runs for 200 turns, accumulates 100K+ token contexts, and expects a responsive experience the whole time. Artificial Analysis just released AA-AgentPerf, the first inference benchmark built specifically for that world.

The gap between benchmarks and reality

The problem with existing hardware benchmarks isn't that they're wrong , it's that they're measuring the wrong thing. Most hardware benchmarks still measure synthetic requests at fixed input and output lengths, with the optimizations production deployments rely on switched off. The question buyers actually need answered , how many agents can this system serve at a speed users will accept? , never appears. And with power becoming the binding constraint on AI infrastructure expansion, that question needs a denominator.

AA-AgentPerf is the first agentic inference benchmark. It replays real coding-agent trajectories against a system under test and finds the maximum number of concurrent agents the system can sustain while meeting market-derived performance targets. Its lead metric is Agents per Megawatt , how many simultaneous agents a system can support per megawatt of measured power, within per-agent output speed and time-to-first-token targets.

What makes the workload different

Agentic inference has a fundamentally different shape than chat. The benchmark captures this with real trajectory data, not synthetic approximations.

• Input lengths range from ~5K to ~131K tokens per request, with a mean of roughly 27K , driven by tool outputs and accumulated history rather than the prompt itself
• Output lengths vary widely across turns: agents mostly emit short tool calls and edits, punctuated by longer stretches of reasoning
• 12+ programming languages are represented, based on the primary language of the source repository
• The test set stays private. Participants receive a representative tuning subset for configuration validation; the full dataset is held out to prevent benchmark-targeted optimization

This shape is what makes agent serving hard, and what uniform-length synthetic benchmarks miss entirely. Tool results balloon the context, outputs are often only a few hundred tokens, and the same prefix comes back turn after turn. A system's KV cache behavior, scheduler, and memory hierarchy decide whether it thrives or collapses under this pattern.