Artificial Analysis' AA-AgentPerf Benchmark Exposes What AI Hardware Really Handles

Every hardware benchmark you've ever used was built for a world that no longer exists. Synthetic prompts, fixed input lengths, no KV cache reuse, no speculative decoding , the numbers look clean, but they describe a deployment style nobody actually ships. Meanwhile, the real workload of 2026 is a coding agent that runs for 200 turns, accumulates 100K+ token contexts, and expects a responsive experience the whole time. Artificial Analysis just released AA-AgentPerf, the first inference benchmark built specifically for that world.

The gap between benchmarks and reality

The problem with existing hardware benchmarks isn't that they're wrong , it's that they're measuring the wrong thing. Most hardware benchmarks still measure synthetic requests at fixed input and output lengths, with the optimizations production deployments rely on switched off. The question buyers actually need answered , how many agents can this system serve at a speed users will accept? , never appears. And with power becoming the binding constraint on AI infrastructure expansion, that question needs a denominator.

AA-AgentPerf is the first agentic inference benchmark. It replays real coding-agent trajectories against a system under test and finds the maximum number of concurrent agents the system can sustain while meeting market-derived performance targets. Its lead metric is Agents per Megawatt , how many simultaneous agents a system can support per megawatt of measured power, within per-agent output speed and time-to-first-token targets.

What makes the workload different

Agentic inference has a fundamentally different shape than chat. The benchmark captures this with real trajectory data, not synthetic approximations.

• Input lengths range from ~5K to ~131K tokens per request, with a mean of roughly 27K , driven by tool outputs and accumulated history rather than the prompt itself
• Output lengths vary widely across turns: agents mostly emit short tool calls and edits, punctuated by longer stretches of reasoning
• 12+ programming languages are represented, based on the primary language of the source repository
• The test set stays private. Participants receive a representative tuning subset for configuration validation; the full dataset is held out to prevent benchmark-targeted optimization

This shape is what makes agent serving hard, and what uniform-length synthetic benchmarks miss entirely. Tool results balloon the context, outputs are often only a few hundred tokens, and the same prefix comes back turn after turn. A system's KV cache behavior, scheduler, and memory hierarchy decide whether it thrives or collapses under this pattern.

Production optimizations are first-class citizens

This is the design decision that separates AA-AgentPerf from everything before it. Three key techniques , which are standard in production but typically banned from benchmarks , are explicitly allowed:

• KV cache reuse: Instead of recomputing the attention keys and values for the entire conversation history on every turn, the system can cache and reuse them. This is critical for long multi-turn sessions where the same prefix appears repeatedly.
• Speculative decoding: A small "draft" model proposes several tokens at once, and the main model verifies them in parallel. When the draft is right (which it often is), you get multiple tokens for the cost of one forward pass.
• Prefill/decode disaggregation: This decouples the computationally intensive prefill stage from the bandwidth-sensitive decode stage, assigning each to dedicated server pools optimized for their distinct resource demands. For agentic workloads with long shared prefixes, this is especially powerful.

Allowing production optimizations makes serving configuration a first-class part of the benchmark. The solution space for serving a model on a given system is wide , inference framework and version, parallelism strategy, speculative decoding setup, disaggregation topology, and dozens of tuning decisions , and it is moving fast. Accuracy verification is included to ensure that an optimization cannot buy capacity at the expense of output quality.

How the measurement works

Rather than reporting peak throughput (which is easy to inflate by sacrificing per-user speed), AA-AgentPerf fixes a service level and asks how far a system can scale while preserving it. Performance targets are derived from Artificial Analysis serverless API benchmarking data , the tiers of service that actually exist in the market today. Speed and latency are measured per request: P25 output speed and P95 time to first token, computed across all requests in a test phase.

At launch, DeepSeek V4 Pro is tested at two tiers:

• Tier 1 (max capacity): 20 tokens/s per agent, P95 TTFT ≤ 10s
• Tier 2 (balanced): 60 tokens/s per agent, P95 TTFT ≤ 5s

For each tier, an exponential ramp followed by a binary search finds the maximum concurrency the system sustains without violating the targets, with metrics computed over steady-state windows where all agents have been active for at least 30 seconds.

The first results: Blackwell dominates, but with caveats

The initial results cover NVIDIA and AMD systems running DeepSeek V4 Pro, at the easiest defined service level (20 tokens/s, 10s TTFT):

System	Configuration	Agents/MW
GB300 NVL72	Rack-scale, disaggregated (6p1d)	61,354
B300 x8	Single node, disaggregated (1p1d)	21,053
MI355X x8	TP8, radix cache, no speculative decoding	3,551
H200 x8	TP8, EAGLE speculative decoding	2,594

Three findings stand out from these numbers:

1. Rack-scale disaggregation is a multiplier, not an increment. The inherent scale advantages, and ability to aggressively disaggregate inference, lead to clear gains in pure compute and Agents per Megawatt vs. single-node. The GB300 NVL72 delivers roughly 3x the power efficiency of a single B300 node , a gap that only exists because rack-scale allows more aggressive prefill/decode splitting across 72 GPUs with 1.8 TB/s NVLink bandwidth.
2. Blackwell is a genuine generational leap over Hopper. The move from Hopper to Blackwell delivers a step change in the number of concurrent agents a system can sustain, in raw capacity and Agents per Megawatt.
3. The AMD numbers are not the AMD ceiling. With the relative infancy of kernel optimizations and config design for DeepSeek V4 Pro on AMD systems, significant improvements in AMD performance are expected in the near-term. Specifically, the MI355X configs are approximately two weeks older than the Blackwell configs and couldn't stably use speculative decoding , a technique that meaningfully boosts the H200 numbers. AMD's measured power draw under heavy load is also well below its rated TDP, suggesting the hardware has headroom that better software hasn't yet unlocked.

Who submitted what

Transparency about configuration provenance is built into the benchmark. In the launch results for DeepSeek V4 Pro, the B300 and GB300 configurations were submitted by NVIDIA, while the H200 and MI355X results run configurations built by Artificial Analysis. Vendor-submitted configs are expected to be more optimized than in-house ones, which is part of why the AMD and H200 numbers should be read as lower bounds rather than final verdicts. Every configuration is published in a public configurations browser.

A live benchmark, open for submissions

AA-AgentPerf is a live benchmark. Submissions of configurations for benchmarking are open now, and results are published on a rolling basis as new hardware, software stacks, and model versions become available. Vendors can also submit improved configurations for systems already on the leaderboard; as serving software advances, published results are updated to reflect how real-world performance evolves.

The roadmap for upcoming versions includes:

• Longer context lengths, extending trajectories up to 1M tokens
• Broader model coverage , gpt-oss-120b results across all submitted systems
• Broader hardware coverage , GB200, B200, H100, MI300X
• Tool execution performance , testing CPU performance for agent tool calls
• Cost normalization , Agents per TCO, cost per task, and $/hr metrics
• Results over time , tracking how each system improves as serving software advances

Hardware vendors and inference providers can reach the team at [email protected] to submit configurations. The full hardware benchmarks dashboard is live now, with results updating on a rolling basis as new submissions come in.

What this changes

The deeper implication here is about what the field has been optimizing for. Benchmarks shape purchasing decisions, which shape what hardware gets built and what software gets optimized. A benchmark that rewards raw token throughput on short prompts will produce an ecosystem tuned for that , not for the multi-turn, long-context, tool-heavy workloads that actually dominate production deployments today.

AA-AgentPerf is also a tool for understanding the impact of each new software and hardware advance, not just each new chip. The fact that configurations for DeepSeek V4 Pro have been improving on a near-daily basis since the model's release means the leaderboard is a living document , and the gap between today's numbers and what's possible with a fully optimized stack is likely still significant for every system on it.