vLLM Rebuilds DeepSeek-V4.1-Flash Serving to Run 58x Faster on Rubin

vLLM and Inferact have ported DeepSeek-V4.1-Flash to a full NVIDIA Vera Rubin NVL72 rack, rebuilding the serving stack around Rubin’s memory domains, matrix instructions, and rack-scale interconnect. Their technical report records a 58-fold gain over GB300 NVL72 at the same interactive service point, with separate paths tuned for high concurrency and single-user latency.

The team measured the system with the AgentX benchmark, which replays traces from agentic applications. The report normalizes aggregate rack output across the 72 GPUs, so throughput uses tokens per second per GPU and latency-sensitive results use tokens per second per user. Prompt and decode disaggregation assigns prompt processing and token generation to separate workers. Wide expert parallelism, or Wide-EP, spreads the model’s mixture-of-experts layers across many GPU ranks. The same methodology underpinned the earlier GB300 evaluation, making the 387-token/s P90 point a rack-to-rack service comparison.

Rubin’s Topology Rewrites the Kernel Plan

• Dual memory domains: Each GPU contains two domains. Streaming multiprocessors read local memory fastest, interleaved allocations more slowly, and the other domain slowest. Placement therefore affects bandwidth-bound operations such as expert-weight reads and the language-model head.
• Wider SM107 matrix instructions: Rubin adds block-scaled matrix multiply-accumulate instructions, including a K64 mixed-precision FP8-by-FP4 operation that performs twice the matrix work of the earlier instruction. MegaMoE and low-latency routed GEMMs use it directly.
• A rack-wide NVLink domain: All 72 GPUs share one NVLink fabric. Eight-way tensor parallelism, known as TP8, can span two compute trays with little additional all-reduce cost.

Rewrites across the mixture-of-experts, attention, matrix multiplication, mHC, and index-selection kernels make prefill steps about 28% faster in the two-way data-and-expert parallel configuration, or DEP2. Corresponding DEP decode steps run about 25% faster.

TP8 Pushes One Session Past 1,800 Tokens per Second

vLLM retained a specialized tensor-parallel decode path for latency-sensitive traffic and rebuilt most of its GB300 implementation for SM107. At concurrency one on AgentX, with DSpark speculative decoding fixed at an average of 3.51 accepted tokens per step, P50 output rose from 1,117 to 1,844 tokens/s per user. The report’s broader end-to-end summary records about 1,810 tokens/s.

• Direct all-reduce publication: GEMM epilogues publish both all-reduce results into a sequence-numbered Lamport mailbox through NVLink multicast, eliminating separate publication kernels.
• Fused receive processing: The receiving kernel sums ranks, applies mHC post-processing, collapses expert routes, computes RMSNorm, and starts the router GEMM before grid-wide synchronization.
• Embedded shared-expert work: Moving shared-expert computation into the routed GEMMs saved 220 microseconds per decode step.

KV Cache Becomes a Rack Resource

Rubin’s higher decode capacity supports enough simultaneous sessions that their live key-value prefixes can exceed the cache attached to an individual rank. A KV cache stores the attention state generated for earlier tokens, allowing subsequent tokens to reuse that work. The GB300 layout pinned each session and its CPU-backed cache to one rank; the Rubin design pools memory across the rack.

1. Mooncake pool in tray DRAM: Any rank can load any cached prefix through Remote Direct Memory Access. Retuning the vLLM scheduler reduced pending lookup work from 6% to 14% of each prefill step to less than 0.5%.
2. Experimental HBM pool: Each prefill rank reserves enough high-bandwidth memory for one maximum-length request, then contributes its remaining cache budget to Mooncake as another storage segment.
3. NVLink-sharded Engram tables: Each prefill instance previously stored two 94.4 GiB n-gram tables in host memory, consuming 378 GiB per tray. The new layout shards their combined data across 18 GPUs, with about 10.5 GiB on each GPU, and fetches peer entries over NVLink through PR #66552.

With prefixes available rack-wide, Dynamo sends each request to the worker and data-parallel rank with the least uncached work. Across three DEP2 instances at 4,800 concurrent sessions, pooling and placement increased served request count by 68%, reduced median time to first token from 29.8 seconds to 4.2 seconds, and raised the cache hit rate from 91.8% to 96.1%. A separate 12-instance simulation reduced P90 time to first token from 0.98 seconds to 0.38 seconds.

The Porting Boundary Moved Up the Stack

As the third installment in vLLM’s DeepSeek-V4.1-Flash series, the Rubin report extends the day-zero bring-up and GB300 scale-out work by identifying the hardware-specific changes required at each layer. Developers can map those findings to three deployment choices:

• High-concurrency throughput: Use P/D disaggregation with Wide-EP and topology-aware DEP kernels.
• Interactive latency: Use the fused TP8 decode path and keep collective communication inside GEMM boundaries.
• Large prefix workloads: Pool KV and Engram data across the rack, then route requests according to uncached work.