Vera Rubin’s AgentPerf Gains Emerge at High Serving Speeds

Artificial Analysis has published the first AA-AgentPerf results for NVIDIA’s Vera Rubin NVL72 rack-scale system. NVIDIA submitted the runs to the third-party benchmark, which replays coding-agent sessions with long, repeated contexts and tool calls. Against the Blackwell Ultra-based GB300 NVL72, Rubin delivered up to 5.7× more total throughput per megawatt on DeepSeek V4 Pro, 5.4× on DeepSeek V4.1 Flash, 3.0× on GLM-5.2, and 2.8× on Kimi K3.

Rubin’s main advantage appears at high per-user generation rates. At each platform’s peak aggregate throughput, its lead ranges from 14% to 45%. As individual sessions run faster, GB300’s efficiency declines much more sharply. That distinction affects how many responsive agents a power-constrained cluster can support.

Agent loops punish the cache

Coding agents repeatedly ask a model to reason, invoke a tool, append the result, and process the expanded conversation. Requests in AA-AgentPerf contain up to 132,000 input tokens, while responses are often short tool calls. On average, 97% of each request’s input repeats the preceding request.

AA-AgentPerf methodology replays trajectories from agents resolving GitHub issues and inserts pauses that simulate tool latency. Three workload properties shape serving performance:

• Large reusable caches: The KV cache stores attention state for previously processed tokens. Thousands of concurrent agents may each retain tens of thousands of cached tokens through tool-call pauses.
• Uneven prefill: Prefill is the work required to process new input before generation begins. One turn may add 10 tokens, while the next may add 69,000 after a tool returns a large file.
• Short outputs: Brief replies give time-to-first-token latency and scheduler overhead a larger share of total request time.

Bandwidth drives the hardware gain

Vera Rubin NVL72 preserves the rack-scale layout of GB300 NVL72, connecting 72 GPUs and 36 CPUs within one NVLink fabric. Its larger gains come from faster memory, interconnects, and low-precision compute.

Both generations provide 288 GB of high-bandwidth memory per GPU. Rubin increases memory bandwidth by 2.4×, NVLink bandwidth by 1.7×, and dense NVFP4 compute by 2.3×. NVFP4 is NVIDIA’s four-bit floating-point format for inference.

The doubled NVLink-C2C bandwidth is especially relevant to agent workloads because it carries KV-cache data between GPU and host memory. Faster transfers let the serving stack retain more active sessions outside GPU memory without imposing the same transfer penalty.

Higher user speed opens the gap

Peak aggregate throughput shows a comparatively narrow generational gain: 1.14× on DeepSeek V4 Pro, 1.45× on DeepSeek V4.1 Flash, 1.35× on GLM-5.2, and 1.26× on Kimi K3. Those figures describe the most efficient point on each system’s throughput curve.

DeepSeek V4 Pro shows how the curves separate under a higher per-user target. Moving from roughly 120 to 165 tokens per second per user reduced GB300’s throughput per megawatt by 81%. Rubin’s declined by 16% over the same range. At the fastest tested rates, the reported per-GPU advantage reached roughly 12× to 18×.

Fastest submitted runs

• DeepSeek V4.1 Flash: 906 tokens/s per user on a single four-GPU server, the leaderboard’s fastest result across all tested models.
• GLM-5.2: 408 tokens/s per user.
• Kimi K3: 255 tokens/s per user on a single eight-GPU vLLM server.

Software keeps session state moving

NVIDIA ran TensorRT-LLM and vLLM behind NVIDIA Dynamo, with settings available in the published configurations. Rubin and GB300 use similar serving designs, making hardware a major source of the measured difference, though kernels and platform-specific tuning remain variables.

• Tiered KV caching: Most Rubin configurations move cached attention state into CPU memory over NVLink-C2C. Transfers overlap with other GPU work, reducing idle time while preserving long agent contexts.
• Session-aware routing: Dynamo sends an agent’s next turn to the worker that already holds its cached prefix. Kimi K3’s linear-attention layers require snapshots because their recurrent state cannot be rolled back like a conventional KV cache.
• Disaggregated serving: Separate workers handle prefill and token generation. A large tool response can then undergo prefill without blocking unrelated agents that are already generating tokens.
• Batch-aware speculative decoding: The server predicts several tokens and verifies them together. NVIDIA tuned draft lengths from seven tokens at low concurrency to three at the highest concurrency, where large batches leave less spare compute.
• Rubin-specific kernels: NVIDIA used tuned NVFP4 kernels for mixture-of-experts layers and fused kernels for model-specific operations such as DeepSeek V4 Pro’s sparse attention.

Limits behind the headline

Rubin’s high-rate DeepSeek V4 Pro configuration returned its first token more slowly. Median time to first token was 1.65 seconds at roughly 165 tokens/s per user, compared with 1.08 seconds for GB300. When the benchmark includes that initial wait in end-to-end speed, GB300 matches or exceeds Rubin’s throughput per megawatt at most tested DeepSeek V4 Pro speeds. Short interactive requests are particularly sensitive to that startup latency.

NVIDIA’s DeepSeek V4.1 Flash runs also capped responses at 2,048 output tokens. Other results for that model on the Artificial Analysis leaderboard used a 16,384-token cap. Artificial Analysis says the shorter limit modestly favors Rubin’s throughput figures for that model.

NVIDIA submitted and tuned every Rubin result. The benchmark and published settings provide external methodology and a path to replication, while the current figures remain vendor-optimized submissions without a reported third-party rerun.

Power efficiency becomes serving capacity

Datacenters often operate within fixed power envelopes. Under that constraint, throughput per megawatt determines the number of active agents the facility can serve. At roughly 165 tokens/s per user, the reported DeepSeek V4 Pro results translate to about five times as many generated tokens on Rubin under the same power budget.

The benchmark applies most directly to coding agents, long-context retrieval-augmented generation systems, and other services dominated by reusable input, short outputs, and pauses between turns. Teams can inspect the published configurations when designing their own deployments, but the results characterize early Rubin kernels and the tested software versions. Later runtime and kernel updates may change both platforms’ performance curves.