vLLM's Kimi K3 Backend Hits 842 Tokens per Second on NVIDIA Blackwell GPUs
A unified CuTe DSL decode path on GB300 pushes Kimi K3 to 842 tokens/s at batch 1, up to 2x faster than stock vLLM.
- vLLM ships a specialized CuTe DSL decode path for Kimi K3 on NVIDIA GB300.
- Hits 842 tokens/s at batch 1 on 16 GB300 GPUs with DSpark speculative decoding, a 1.73x speedup.
- Delivers 1.5-2x higher decode throughput across TP8 and TP16 configurations without speculation.
- Posts 1.8x higher P90 interactivity on SemiAnalysis AgentX, 403 vs 224 tokens/s/user.
- Cuts per-layer kernel count from 23 to 13 and runtime from 130 to 69 microseconds.
- Introduces a fast-track RFC for agent-generated optimizations with isolated scope and reproducible evals.
vLLM has added a specialized decode backend for Moonshot’s Kimi K3 on NVIDIA GB300 systems. Built with NVIDIA, the backend uses custom CuTe DSL kernels to fuse attention, mixture-of-experts computation, and inter-GPU communication. The narrower execution path cuts kernel launches and raises token-generation throughput on supported Blackwell configurations.
Decode generates tokens after a model has processed the prompt, so its latency directly affects interactive workloads such as coding agents. According to vLLM’s report, the backend exceeds 840 tokens per second at batch 1 across 16 GB300 GPUs with DSpark and a synthetic acceptance length of six tokens. It also delivers 1.5 to 2 times higher decode throughput in several configurations and up to 1.8 times higher P90 interactivity on the AgentX workload.
Kernels collapse the decode loop
The backend replaces most operations in Kimi K3’s decode loop with kernels written in CuTe DSL, NVIDIA’s low-level GPU programming system. For one model layer, the standard vLLM path takes 130 microseconds across 23 kernel launches. The specialized path takes 69 microseconds across 13 launches by combining operations and coordinating work across CUDA streams.
Four changes drive the gains
- Shape-aware matrix multiplication: Small-matrix kernels handle attention input projections at batch sizes 1 and 2. Larger batches use split-K kernels, which divide one matrix multiplication among more GPU workers.
- Communication fused with computation: The attention output projection writes partial results into NVLink-accessible buffers. Receiving kernels perform the all-reduce, combine those results across GPUs, and continue into the residual computation.
- Sharded MoE routing: Tensor-parallel ranks share the work of computing router scores and latent input projections. The shared expert runs on a separate stream, allowing it to overlap with routing or routed-expert computation.
- Weight prefetching: A side-stream kernel loads weights while communication is using less than the available memory bandwidth. The next operation can then retrieve more of its weights from cache.
Throughput holds across the tested range
The specialized backend improves decode throughput across the reported batch sizes, including runs without speculative decoding. TP8 and TP16 refer to tensor parallelism across 8 and 16 GPUs, respectively.
| Configuration | Standard vLLM | Specialized backend | Gain |
|---|---|---|---|
| TP16, batch 1 | 136 tokens/s | 256 tokens/s | 1.88× |
| TP16, batches 16–64 | Varies by batch size | 1.62–1.71× | |
| TP8, batches 16–64 | Varies by batch size | 1.31–1.43× | |
| TP16, batch 1, DSpark AL=6 | 488 tokens/s | 842 tokens/s | 1.73× |
SemiAnalysis’s AgentX benchmark replays coding-agent traffic with long contexts, multiple conversation turns, tool pauses, parallel subagents, and KV-cache reuse. At concurrency 1, the optimized build reaches 403.2 tokens per second per user at P90 interactivity, compared with 223.7 for the standard path.
A narrow deployment lane
The optimization targets Kimi K3 decode workloads on 8 or 16 NVIDIA Blackwell GPUs within one NVLink domain that supports multicast. It covers decode batches from 1 through 64 and supports optional decode context parallelism for long sequences. Unsupported configurations fall back to the standard vLLM path.
| Area | Supported scope |
|---|---|
| Hardware | NVIDIA Blackwell, including GB300 |
| GPU count | 8 or 16 GPUs |
| Topology | One NVLink domain with multicast support |
| Decode batch size | 1–64 |
| Long contexts | Optional decode context parallelism |
| Serving features | Full-context serving, temperature sampling, structured output, and speculative decoding |
The backend accelerates token generation rather than prompt prefill. Its changes remain within model-specific decode code, leaving shared input preparation, sampling, and KV-cache management untouched. Deployments can select the specialized path with the backend value latency_specialized, subject to the supported hardware and topology.
Agent-written kernels face evidence gates
The release also exercises a proposed fast-track RFC for agent-generated optimization work. Because fused CuTe kernels are difficult to validate through line-by-line review alone, the proposal requires contributors to isolate changes within a model’s code, define supported configurations, and provide reproducible correctness and performance results before design review.
An audit of the Kimi K3 kernels found that the compiler could move activation reads ahead of a synchronization wait. The fix limited that load pattern to static weights. Long-context evaluations, including MRCR and AIME25, also exposed numerical problems in attention and communication kernels that unit tests using mock inputs had missed.
The pattern spreads to other models
A DeepSeek PR applies the same opt-in decode_backend: latency_specialized configuration to DeepSeek-V4.1-Flash. That implementation fuses all-reduce operations at each sublayer boundary and captures per-step CPU work in a CUDA graph for replay.
For Meta’s Muse Spark Open, the team reports preliminary throughput above 950 tokens per second per user with seven draft tokens. GLM 5.3 is also planned for the specialized path.
For Kimi K3 deployments using TP8 or TP16 within a supported NVLink domain, the backend offers a model-specific route to lower decode latency without changing shared vLLM subsystems. The accompanying RFC defines how similar agent-generated kernels could enter the serving stack through constrained scope, explicit fallbacks, and reproducible validation.
Comments
