SGLang Breaks 12,000 Tokens per Second on DeepSeek V4 Pro

SGLang just hit a new inference record: over 12,000 output tokens per second per GPU on DeepSeek V4 Pro, the 1.6 trillion parameter MoE model, running on NVIDIA's GB300 NVL72 rack. The result was validated by SemiAnalysis's InferenceX benchmark suite and combines three separate performance levers: the GB300's Blackwell Ultra hardware, NVIDIA Dynamo for disaggregated serving, and Multi-Token Prediction (MTP) speculative decoding. It's a meaningful milestone, and the story behind it is more interesting than the headline number.

The Model, the Machine, and the Stack

DeepSeek V4 Pro is a 1.6 trillion total parameter MoE model with roughly 49 billion active parameters per token, a 1 million token context window, and MIT licensing. Despite the astronomical parameter count, only a fraction of the model activates per token, which is what makes MoE models uniquely suited to large-scale, high-throughput inference. V4 Pro features a hybrid attention design that combines Compressed Sparse Attention (CSA) with a new Heavily Compressed Attention (HCA) head for cheap long-context prefill.

The hardware underneath this benchmark is NVIDIA's GB300 NVL72, the current top of the Blackwell Ultra lineup. The GB300 NVL72 is the most powerful platform for LLM inference in the Blackwell family. What makes it especially well-suited for MoE models is the rack-scale NVLink domain: 72 GPUs are connected into a single high-bandwidth fabric, which keeps the all-to-all communication that MoE expert dispatch requires entirely on fast NVLink rather than spilling onto slower InfiniBand.

Three Levers, One Record

The 12K tokens/GPU/second figure is not the result of a single trick. It stacks three distinct optimizations on top of each other:

• NVFP4 quantization. Running MoE expert weights in FP4 precision reduces memory bandwidth pressure, halves the communication traffic for token dispatch, and taps into the GB300's 1.5x higher FP4 Tensor Core throughput compared to the previous Blackwell generation. It also frees HBM capacity for a larger KV cache, enabling higher concurrency.
• NVIDIA Dynamo for disaggregated inference. Dynamo is the orchestration layer above inference engines; it doesn't replace SGLang, it turns it into a coordinated multi-node inference system. Dynamo allows for disaggregated serving, which splits the prefill and decode stages of inferencing among multiple GPUs to optimize resource utilization and drive down the cost of tokens. In practice, SGLang's integration with Dynamo couples its KV-aware router with SGLang's own radix tree cache, enabling flexible KV cache transfer between prefill and decode workers.
• Multi-Token Prediction (MTP). MTP is a form of speculative decoding where the model predicts multiple future tokens in a single forward pass, then verifies them. It exploits the fact that small-batch decode is memory-bandwidth-bound, not compute-bound, so there's spare compute capacity to draft extra tokens cheaply. Enabling MTP on GB300 NVL72 was specifically called out in the SGLang-NVIDIA roadmap as a key lever for unlocking further performance gains.

How We Got Here: A Software Sprint

The 12K number did not appear on day one. The cost per million output tokens for the GB300 reaches $0.156 at 50 tokens/second/user, assuming 8K tokens input and 1K tokens output. Getting there required weeks of iterative optimization tracked publicly by SemiAnalysis's InferenceX benchmark suite. InferenceX is a continuously running benchmark suite that evaluates real-world inference performance across popular open-source frameworks and models on hundreds of accelerators.

The GB300 SGLang trajectory is a case study in how much software matters. The most dramatic jump came from the implementation of W4A4 (MXFP4) MegaMoE, a new fused MoE kernel that DeepSeek shipped alongside V4 Pro. MegaMoE splits experts into waves and schedules each wave separately, allowing finer-grained overlapping of dispatch, compute, and combine operations. The paper claims a theoretical 1.92x speedup over the naive kernel, which implies the naive implementation was spending close to half its time on communication alone. Beyond kernels, widening decode expert parallelism from EP=8 to EP=16 and scaling prefill workers from one or two per decode worker to four to twelve was the main architectural lever that pushed GB300 throughput to its current level.

The SGLang team has worked closely with NVIDIA across multiple GPU generations to unlock step-function gains in inference performance for large-scale MoE deployments, building on prior results that delivered 4x speedups on Blackwell B200 versus Hopper H200. The GB300 result extends that trajectory to Blackwell Ultra.

The Interactivity Curve: Where It Shines and Where It Doesn't

The 12K figure is a peak throughput number at high concurrency. The more useful way to read this benchmark is the full throughput-interactivity curve, which plots tokens/GPU/second against tokens/second/user (a proxy for response latency). LMSYS specifically notes that performance stays strong across the entire interactivity curve, which matters because peak throughput at low interactivity is easy to achieve but useless for real deployments.

• High-throughput batch workloads (offline summarization, batch coding, document processing): this is where GB300 + SGLang + MTP is unambiguously best-in-class across all hardware tested.
• Interactive workloads (chat, agentic loops, coding assistants): MTP helps significantly here because small-batch decode has spare compute capacity. The GB300's 72-GPU NVLink domain keeps expert parallelism wide enough to maintain low latency even at high concurrency.
• Single-node deployments: B200 and B300 in 8-GPU NVLink islands scaled out over InfiniBand hit the expert parallelism wall much earlier. The GB300 rack-scale advantage is fundamentally a scale-up domain story, and it requires the full NVL72 configuration to realize.

The Competitive Landscape

This benchmark lands in a context where inference performance has become a primary competitive axis. In recent industry benchmarks, Dynamo boosted the inference performance of NVIDIA Blackwell GPUs by up to 7x, lowering token cost and increasing revenue opportunity for millions of GPUs with free, open-source software. SGLang now powers inference across more than 400,000 GPUs worldwide, generating trillions of tokens in production daily.

AMD's MI355X has made real progress on DeepSeek V4 Pro since launch. The AMD SGLang engineering team massively improved performance in the first month, achieving more than 100x improvement in throughput by Day 26. But the GB300 NVL72 with SGLang and MTP remains unbeatable across the full interactivity curve as of the latest InferenceX data. The gap is structural: it comes from the 72-GPU NVLink domain enabling wider expert parallelism than any 8-GPU island can match.

Huawei's Ascend 950DT is the other notable entrant. Unlike AMD, DeepSeek V4 dropped with full open weights and Huawei had Day 0 support, making it one of only two stacks (alongside NVIDIA CUDA) that worked out of the box at launch. The CANN stack's performance trajectory will be worth watching, particularly for China-based deployments where CUDA hardware is export-restricted.

What This Means for Deployment

For teams running DeepSeek V4 Pro in production, the practical takeaways are:

• SGLang offers three primary serving recipes for DeepSeek V4 on NVIDIA Blackwell and Hopper, each tuned for a different latency/throughput profile: low-latency, balanced, and max-throughput, along with specialized recipes for long-context workloads and prefill/decode disaggregation.
• MTP is now the default recommendation for interactive workloads on Blackwell. It was available on Day 3 post-launch from SGLang and delivers meaningful per-user throughput gains without model quality loss.
• The GB300 NVL72 is the right hardware if you need maximum throughput on the full 1.6T model. For V4-Pro, 8x HGX B300 is the cleanest single-node deployment option, with native FP4 execution, full 1M context, and no model-length cap. The GB300 NVL72 rack goes further but requires the full rack-scale commitment.
• V4 requires vLLM version 0.7.0 or SGLang version 0.4.4 or later.

The broader implication is that the inference software stack is now a first-class competitive differentiator. The same hardware running different software can produce 8x different throughput numbers, as the GB200 NVL72 trajectory demonstrated over four months. The race is no longer just about who ships the fastest chip; it's about who ships the fastest chip with the best software to extract it.