Inferact's Kimi K3.1 Pro DSpark Accepts 23% More Tokens, Cuts Draft Memory
Inferact and Red Hat released a new DSpark draft model for Kimi K3.1 Pro. It gets up to 30% more draft tokens accepted than the preview, and its draft memory stays capped even at 1M-token context.
- Inferact and Red Hat released a new Kimi K3.1 Pro DSpark draft model that replaces the preview.
- Acceptance length is up to 30% higher on HumanEval and 23% higher pooled across benchmarks.
- Every draft layer uses a 2K sliding window, so draft KV memory stays fixed up to 1M-token context.
- At 1M context, acceptance rose from 2.77 to 3.42 compared with the full-attention preview.
- Weights are free. Upgrading is a one-line
--speculative-configchange in vLLM, and the vLLM recipes include a preset. - Gains are weakest on chat and agentic coding (MT-Bench 3.39, SWE-bench Pro 3.30).
Inferact and the Red Hat AI Speculators team have released a new DSpark checkpoint for Kimi K3.1 Pro. It replaces Kimi-K3.1-Pro-DSpark-preview, accepts more speculative tokens across every shared benchmark, and bounds the draft model’s KV cache with a 2,048-token attention window.
A small draft model makes speculative decoding practical for Kimi K3.1 Pro, a 2.8-trillion-parameter mixture-of-experts model. The draft proposes several tokens, then the target model verifies the block in one forward pass. Longer accepted runs reduce the number of expensive target-model passes required to generate a response. Compatible rejection sampling preserves the target model’s sampling distribution.
Seven guesses, longer accepted runs
DSpark reads hidden states the target model has already computed and proposes a token block in one parallel pass. The target then verifies those candidates together. Acceptance length measures how many output tokens generation advances per verification step, on average.
A block configured with seven speculative tokens has an acceptance-length ceiling of eight: seven accepted draft tokens plus one token produced by the target. Higher acceptance usually improves decoding speed, although the final gain depends on hardware, concurrency, batching, and draft overhead.
| Benchmark | Preview | New checkpoint | Change |
|---|---|---|---|
| HumanEval | 3.585 | 4.660 | +30% |
| GSM8K | 4.597 | 5.566 | +21% |
| SWE-bench Pro | 2.741 | 3.302 | +20% |
| BEAM 1M | 2.765 | 3.421 | +24% |
| Pooled result | 3.154 | 3.864 | +23% |
The published evaluations used sampling at temperature 1.0 with top_p=0.95, avoiding greedy-decoding best cases. BEAM 1M covered prompts ranging from 826,000 to 1.04 million tokens.
The new draft is a five-layer dense decoder with 2.5 billion parameters stored in BF16, down from 2.7 billion in the preview. That reduction saves roughly 0.4 GB of raw BF16 weight storage before runtime overhead.
A 2K window caps draft memory
The preview used full causal attention across the available context, causing its KV cache to grow with every token. At contexts approaching one million tokens, that cache competed with the target model for GPU memory.
The new checkpoint combines grouped-query attention with a 2,048-token sliding window in every layer. Once the window fills, each draft layer retains keys and values for only the latest 2,048 positions. Per-step attention work and draft KV-cache usage therefore remain bounded as the sequence grows.
Long-context acceptance also improved: BEAM 1M rose from 2.765 to 3.421. The recent target hidden states consumed by DSpark already encode information from the full prompt, so bounded draft attention can retain useful long-range signals without storing the entire draft history.
vLLM’s redesigned KV-cache management provides the serving support required by this architecture. It handles a sliding-window draft across different parallelism layouts and prefill/decode disaggregation, where prompt processing and token generation run on separate GPU pools.
From checkpoint to vLLM
Kimi K3.1 Pro is a multimodal mixture-of-experts model with 2.8 trillion total parameters, 104 billion active parameters per token, and a one-million-token context window. The mixture-of-experts design routes each token through a subset of the model, reducing computation while leaving the full weight set distributed across the deployment. Reference configurations use eight GB300 GPUs.
The weights are available from Inferact and a Red Hat mirror. Existing preview deployments can switch the draft model name in their speculative configuration:
vllm serve moonshotai/Kimi-K3.1-Pro \
--trust-remote-code \
--tensor-parallel-size 8 \
--reasoning-parser kimi_k3 \
--speculative-config '{"method":"dspark",
"model":"Inferact/Kimi-K3.1-Pro-DSpark",
"num_speculative_tokens":7,
"draft_sample_method":"probabilistic",
"rejection_sample_method":"block"}'The official vLLM recipes use seven speculative tokens for one concurrent client and four once concurrency reaches 20. Low-concurrency requests tend to benefit most because speculative decoding reduces memory-bound target passes. Larger batches keep the target GPUs busier, allowing draft overhead to consume more of the gain.
- Pin revisions: The command enables
--trust-remote-code, which executes repository code. Production deployments should review and pin model and code revisions. - Tune block length: Start with seven tokens at low concurrency and four at higher concurrency, then benchmark against representative traffic.
- Track acceptance by position: A high average can conceal weak acceptance for later guesses and wasted draft computation.
- Measure both latency and throughput: Acceptance length is a useful proxy, while end-to-end gains depend on the complete serving stack.
Agent loops accept fewer guesses
Acceptance varies by workload because structured continuations are easier to predict several tokens ahead than open-ended conversation or agentic code edits.
| Workload | Acceptance length | Output pattern |
|---|---|---|
| GSM8K | 5.566 | Structured math reasoning |
| HumanEval | 4.660 | Short code generation |
| BEAM 1M | 3.421 | Long-context generation |
| MT-Bench | 3.393 | Open-ended conversation |
| SWE-bench Pro | 3.302 | Agentic software engineering |
On SWE-bench Pro, only 9% of candidates at the seventh draft position are accepted. Shorter speculative blocks can save compute on agentic workloads where acceptance falls sharply with each additional position.
A recipe pull request also documents a tool-calling configuration on vLLM’s main branch that failed to start with either DSpark checkpoint pending additional fixes. Some recipe presets retain the preview checkpoint to reproduce earlier benchmark runs.
Measure the speedup on your stack
The preview previously reached a 3.0-times decode speedup at batch size one on GSM8K. The new checkpoint’s higher acceptance should reduce target verification work under comparable conditions, while its sliding window removes context-length growth from the draft cache. Actual latency gains still depend on request mix, concurrency, GPU topology, and vLLM configuration.
Kimi users also have a draft from LithosAI for lossless speculative decoding with Kimi K3.1 Pro, with support for SGLang and vLLM. Engine compatibility may determine which implementation fits an existing serving stack.
Teams already serving Kimi K3.1 Pro with the preview can upgrade by changing the checkpoint name and retuning the speculative-token count. Low-concurrency coding, mathematical reasoning, and long-context generation offer the strongest initial targets. High-concurrency chat and agentic tool use require workload-specific benchmarks and often benefit from shorter draft blocks.