vLLM lifts Kimi K3 throughput by up to 2.8×

vLLM has accelerated serving for Moonshot AI’s Kimi K3 through changes to scheduling, recurrent-state management, mixture-of-experts kernels, speculative decoding, and parallelism. In reported tests, the optimized build delivered 2.2× to 2.8× the throughput of v0.27.1 while cutting time to first token by 72% to 85%. The vLLM benchmark post documents the work.

The comparison used identical model weights and hardware, so the gains came from serving software. That makes the results relevant to teams operating Kimi K3 and to inference-engine developers working on other recurrent mixture-of-experts models.

One node, 2.8× more throughput

The benchmark used an 8K-token input, a 1K-token output, tensor parallelism across eight ranks, and a DSpark speculator proposing up to eight draft tokens per step. Tests ran on an NVIDIA B300 node with CUDA 13.3 at concurrency levels of 1, 4, and 16. Concurrency represents the number of simultaneous requests.

These controlled results cover one workload shape and hardware configuration. Production gains will vary with prompt length, output length, request arrival patterns, cache-hit rates, memory pressure, and the selected parallelism strategy.

Kimi K3 stresses the whole stack

Moonshot’s architecture combines attention with KDA recurrent layers and LatentMoE experts. Recurrent layers update a persistent state as tokens arrive, while mixture-of-experts layers route each token through a subset of available experts. Kimi K3 also uses low-precision MXFP4 expert kernels and supports speculative decoding, which drafts several tokens before the main model verifies them.

That combination creates bottlenecks outside the large matrix multiplications that dominate conventional Transformer inference. Scheduler token budgets can split a prefill into extra forward passes. Prefix caching must preserve recurrent state. Draft rejection requires exact rollback. Small indexing operations accumulate across layers, while tensor and pipeline parallelism add communication and memory constraints.

Four bottlenecks, four targeted fixes

The team measured several changes independently across the broader optimization effort. Their effects overlap and should not be added together. The headline comparison also disabled prefix caching, so the KDA caching result applies to configurations that enable that feature.

The scheduler stops wasting its token budget

The previous scheduler reserved speculative-token capacity across the maximum sequence count, even when only one or a few requests were active. Much of max_num_batched_tokens could remain unused, causing a single long prompt to span multiple forward passes.

An adaptive scheduled-token budget lets the active batch use more of the available capacity. vLLM also raised the default max_num_batched_tokens value from 8,192 to 16,384 on high-memory GPUs, allowing the tested 8K prefill to complete with less scheduler fragmentation.

KDA checkpoints stay inside the prefill

KDA prefix caching previously split a prefill at the final cacheable block boundary. A short suffix could then trigger another full-model pass through attention, routing, MoE computation, and tensor-parallel collectives.

The revised path exports the recurrent-state checkpoint during the original prefill pass. A follow-up change supports partial prefix-cache hits alongside speculative decoding, reducing duplicate computation when only part of a prompt matches cached state.

Small tensor copies leave the hot path

Batches containing both speculative and ordinary requests previously ran six index_select operations and two index_copy operations per layer. Replacing them with contiguous zero-copy slices and direct output writes improved throughput at concurrency 4 and 16. Batch size 1 showed no measurable change because mixed-request bookkeeping was absent.

MoE finalization moves into the kernel

The MXFP4 expert path previously finalized top-k routing in a separate operation. Fusing that work into the latent-tail kernel removed one kernel launch and one intermediate tensor write-and-read cycle, reducing end-to-end latency by roughly 5%.

ReplaySSM shrinks rollback state

Speculative decoding requires the serving engine to recover the exact recurrent state when the main model rejects drafted tokens. A direct implementation stores KDA state at every draft position, producing T additional state writes when each step proposes T tokens.

ReplaySSM keeps recent state-space-model inputs in a buffer and reconstructs the accepted state at commit time. Rejecting a draft only moves the buffer pointer. The design exchanges limited recomputation for fewer state writes and lower cache consumption.

With the same 46.48 GiB cache budget, the post reports about 11% more effective capacity under tensor parallelism size 8, with no measured accuracy loss. The technique addresses a recurring cost of combining token speculation with recurrent layers.

DCP spreads long contexts across ranks

Tensor parallelism replicates Kimi K3’s multi-head latent-attention KV cache on every rank. Additional ranks increase compute capacity, but each rank still holds the same context data, limiting the total number of long sequences that fit in memory.

DCP results show how decode context parallelism shards that cache along the sequence dimension. The published workload used a 114K-token shared prefix, a 6K-token suffix, and 400 generated tokens.

The DCP8 run completed with zero request errors. The close GSM8K scores serve as a quality check; the benchmark does not establish that context parallelism improves model accuracy.

Reproduce the published run

The published server configuration uses fastsafetensors loading, Kimi K3 reasoning and tool-call parsers, tensor parallelism size 8, and the eight-token DSpark speculator:

vllm serve moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.9 \
  --reasoning-parser kimi_k3 \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --max-model-len auto \
  --no-enable-prefix-caching \
  --speculative-config '{"model":"RedHatAI/Kimi-K3-speculator.dspark","method":"dspark","num_speculative_tokens":8,"draft_sample_method":"probabilistic","rejection_sample_method":"standard"}'

Controls that affect the comparison

• Prefix caching: Both headline runs disabled it because v0.27.1 had a known Kimi K3 caching issue that was fixed later. Workloads with caching enabled can produce different results.
• Build version: Pin the optimized vLLM release or commit used for testing because the CLI command does not encode a software version.
• Remote code: --trust-remote-code executes code from the model repository. Review and pin the repository revision before using it in a controlled environment.
• Hardware and runtime: The reported numbers came from a B300 node with CUDA 13.3. GPU generation, drivers, CUDA libraries, and available memory affect kernel selection and throughput.
• Traffic shape: Match prompt length, output length, concurrency, speculative-token count, and memory utilization before comparing results.

What carries over to other models

• Broadly reusable: Adaptive scheduler budgets, copy elimination, kernel fusion, and sequence-sharded KV caches can benefit other inference workloads.
• Architecture-dependent: In-pass recurrent checkpoints and ReplaySSM apply to models with Mamba-style or related recurrent state.
• Kimi-specific: The reasoning parser, tool-call parser, KDA state layout, and LatentMoE MXFP4 kernels depend on Kimi K3’s implementation.

The wider effort is documented in the optimization tracker. Community contributions include DeepEPv2 communication with DeepGEMM MXFP4 support and decode context parallelism, while Thien Tran developed sequence-parallel GEMM paths.

Existing Kimi K3 deployments on vLLM v0.27.1 have a clear reason to benchmark a newer build under production traffic. For workloads dominated by long shared prefixes, DCP may provide the larger capacity gain; for short or lightly concurrent requests, scheduler and prefill changes are more likely to dominate.