vLLM Matches DeepSeek's DeepInfer and Scales 7.9x Across 72 GPUs
vLLM matches DeepSeek's own inference engine on V4.1-Flash and scales to 954K tokens/s per GPU on a GB300 NVL72 rack.
- vLLM matches DeepInfer decode within 6% across batch sizes 8-256 on DeepSeek-V4.1-Flash.
- Hits 954K tokens/s per GPU on GB300 NVL72, 7.9x single-node baseline, at $38.96/GPU-hour.
- Specialized decode path delivers 826 tokens/s for a single user on four B300 GPUs.
- Three prefill PRs stack to a 35-44% throughput gain on new tokens after 128K cache hits.
- EPLB raises MoE routing balance from 0.35 to 0.94, recovering 97% of uniform-routing headroom.
- Full engineering writeup covers DEP2 prefill, Wide-EP decode, and CPU overhead fixes.
vLLM has come within 6% of DeepInfer’s decode throughput across batch sizes 8 through 256, according to its published benchmark for DeepSeek-V4.1-Flash. Its prefill performance also follows DeepInfer’s published curve.
A general-purpose engine reaching that performance gives teams a path to serve the model without maintaining a separate runtime. DeepInfer, which DeepSeek built from vLLM, targets this model in offline mode. vLLM adds online scheduling, multi-model support, prefill and decode disaggregation, and rack-scale expert parallelism.
Near parity on one node, 7.9× at rack scale
The published results separate engine-level comparisons from online serving across an NVIDIA GB300 NVL72, a rack system connecting 72 GPUs through NVLink.
| Test | Result | What it measures |
|---|---|---|
| Decode comparison | Within 6% of DeepInfer | Batch sizes from 8 through 256 |
| Prefill comparison | Tracks DeepInfer’s curve | Prompt-processing throughput |
| Rack-scale AgentX | 954K tokens/s/GPU | Online serving with prefill/decode disaggregation and Wide-EP |
| Per-user rate | P90 of 196 tokens/s | Reported user-level throughput on AgentX |
| Single-user path | 826 tokens/s | Configuration tuned for low latency |
| Scaling | 7.9× the single-node result | Throughput gained by expanding across the rack |
| API-price equivalent | $38.96 per GPU-hour | Throughput valued at DeepSeek’s off-peak API rates |
SemiAnalysis AgentX represents agent-style traffic with long, growing contexts. That workload exercises request scheduling, KV-cache movement, expert routing, and concurrent decoding in addition to raw kernel speed.
Shared kernels close the gap
DeepInfer began as a vLLM-based engine, while vLLM has continued integrating DeepSeek’s low-level GPU routines from MegaAttn, MegaMoE, and Mega-mHC. Both engines therefore execute many of the same optimized attention and Mixture-of-Experts operations.
vLLM’s serving layer extends those kernels across separate prefill and decode pools. Prefill processes the input prompt and builds its key-value cache; decode generates subsequent tokens from that cache. Assigning each phase to its own GPU pool allows independent tuning for compute, memory, batching, and latency.
Wide expert parallelism, or Wide-EP, distributes a Mixture-of-Experts model’s experts across many GPUs. Because each token activates only a subset of those experts, wider distribution reduces the amount of expert-weight data each rank must read during a decode step.
Memory math sets the parallelism
DeepSeek-V4.1-Flash uses only 890 bytes per token for its global KV cache because it combines cross-layer KV sharing with FP4 compression. That compact cache leaves more memory available for long contexts and batching.
MoE weight reads account for most decode time, while attention contributes relatively little. Context growth also has a limited effect on each step: increasing the context from 128K to 855K tokens, a 6.7× expansion, adds 0.6 ms, or 7%, to model decode latency.
| Decision | Evidence | Effect |
|---|---|---|
| Use DEP2 for prefill | DEP4 reached 53.3K tokens/s/GPU, 83% of DEP2’s 64.4K | Fewer ranks reduce load imbalance while retaining enough KV capacity |
| Expand decode to DEP32 | MoE time fell from 13.4 ms at DEP4 to 6.0 ms at DEP32 | More GPUs reduce expert-weight reads per rank |
| Simplify decode sweeps | Step time changed little between 128K and 855K contexts | Benchmarks could avoid replaying complete agent traces for every configuration |
DEP combines data-parallel attention with expert parallelism, and the number indicates the participating GPU count. A DEP2 prefill instance spans two GPUs; a DEP32 decode instance spans 32.
Three prefill changes compound
At 32K newly processed tokens, three changes increased prefill throughput by reducing repeated indexer work and memory-address translation overhead:
- Extended bounded replay through layer 20. The indexer skips computation for all except the final 128 tokens, improving throughput by 19.9%.
- Mapped shared host tables with 2 MB GPU pages. Larger pages reduce misses in the cache that translates virtual addresses, adding 2.8%.
- Used DeepSelect top-k in the prefill indexer. Faster candidate selection adds another 15.6%.
Together, the changes raise new-token prefill throughput by 35.6% to 43.9% across workloads adding 4K to 64K tokens to a cached 128K prefix. Throughput for a 64K prefill rises from 71.2K to 102.5K tokens/s/GPU.
Balanced experts expose the CPU bottleneck
vLLM’s Expert Parallel Load Balancer periodically rearranges expert placement according to observed demand. On a scale where 1.0 represents uniform work across ranks, EPLB raises balancedness from 0.35 to 0.94 for the target model and from 0.21 to 0.89 for the draft model.
Speculative decoding uses the smaller draft model to propose tokens and the target model to verify them. Better expert placement recovers about 97% of the target model’s gap to uniform routing and 93% of the draft model’s gap. A new NVLink backend rebalances a DEP32 instance in under one second without pausing token generation.
Shorter GPU steps made host-side scheduling and synchronization the next constraint. An untuned DEP32 step took 17.57 ms, compared with 13.84 ms for DEP16, even though GPU execution was 1.1 ms faster at DEP32. CPU work consumed the hardware gain.
- KV transfer: A NIXL connector raised transfer throughput from 141K to 440K tokens/s/GPU.
- Rank synchronization: A recursive-doubling CPU all-reduce replaced Gloo.
- Cache admission: Each step accepts at most four new KV-cache loads, limiting bursts of host work.
- Request admission: Processing time for an 855K-token prompt fell from 38.8 ms to 6.0 ms.
The deployment case, and its limits
Teams serving DeepSeek-V4.1-Flash can use upstream vLLM for online, rack-scale deployment while retaining performance close to DeepInfer’s model-specific offline runtime. The same build supplies scheduling, disaggregated serving, multi-model support, and Wide-EP across an NVL72 rack.
The techniques apply most directly to large MoE models whose decode time is dominated by expert-weight reads. Actual gains will depend on model architecture, routing skew, prompt distribution, concurrency, interconnect topology, and CPU capacity, so production validation should reproduce the target workload and latency objectives.
Comments
