vLLM's DSpark Beats Every Fixed-Length Config on DeepSeek-V4 at Any Load

Speculative decoding is one of the most powerful tricks in the LLM serving playbook: a cheap draft model proposes several tokens at once, and the full target model verifies them all in a single forward pass. The target model accepts the longest prefix consistent with its own distribution and appends one bonus token, accelerating generation without any quality loss. The catch has always been tuning the draft length. Too short, and you leave speed on the table at low load. Too long, and at high concurrency you're burning compute verifying tokens that will almost certainly be rejected.

DSpark, DeepSeek's speculative decoding framework, introduced a confidence head that scores each drafted token's probability of surviving verification. While parallel drafters efficiently propose long token sequences in a single forward pass, they suffer from rapid acceptance decay due to a lack of inter-token dependencies, and indiscriminately verifying these extended blocks wastes critical batch capacity on tokens with high rejection risks. vLLM now closes that gap with adaptive verification: instead of committing to a fixed draft length at deploy time, the engine decides per step how many tokens are worth verifying, based on live confidence scores.

Why the old approach breaks at scale

The acceptance rate of draft tokens is not flat across positions. On DeepSeek-V4-Pro-0813, the first token of a 7-token draft survives verification more than 70% of the time. The last one, less than 10%. Under the old fixed-length scheme, you had to pick one number for your whole deployment and live with it. No static num_speculative_tokens is optimal across concurrencies: the crossover moves with load and workload-dependent acceptance rates.

A uniform K, however well chosen, pays for this within-batch spread either in wasted verification (K too long for cold requests) or lost acceptance (K too short for hot ones). In practice, teams running DeepSeek-V4 at scale had to benchmark their typical traffic shape and pick a static number, then re-tune whenever load patterns changed.

How adaptive verification works

DSpark unifies high-throughput parallel generation with adaptive, load-aware verification through a semi-autoregressive architecture that couples a parallel backbone with a lightweight sequential module. The key piece is the confidence head: a single lightweight linear projection that, for each draft position, outputs a probability that the target model will accept that token.

Those per-position scores are turned into survival probabilities: the running product of acceptance chances along each request's draft. The scheduler then solves a simple optimization: given a total verification budget B (tokens the GPU can afford to verify this step), allocate those slots to the highest-survival draft positions across all requests in the batch.

• Slots compete globally across requests. Position 5 of a highly confident request can outrank position 1 of a low-confidence one.
• The budget B is computed on the CPU while the GPU is still executing the previous step, using a profiled cost table built at startup.
• The per-request allocation runs on the GPU against current confidence values, compiled to Triton via torch.compile, with no host readback.
• Variable-length verification is handled by varlen decode CUDA graphs, enabled by DeepSeek's open-sourced varlen indexer kernel from DeepGEMM.

The cost model itself is a lookup table profiled at startup: the engine times dummy steps across all CUDA graph shapes, takes the median of five runs per shape, and uses that to estimate the cost of any given token count. Inside captured CUDA graphs, cost is a staircase rather than a line because of CUDA graph padding, and the budget algorithm is strongly encouraged to stay within the CUDA graph region.

One config, every load level

With adaptive verification on (num_speculative_tokens: 7), speculative decoding holds the Pareto frontier from concurrency 1 to 256 on 8×B300, behaving like a long fixed block at low concurrency and a short one at high concurrency. That means you get both benefits without knowing the shape of your workload in advance.

Benchmarks were run on DeepSeek-V4-Pro-0813 with TP=8 on 8×B300 GPUs, FP8 KV cache, expert parallelism, and 880 prompts swept over concurrencies from 1 to 256. The adaptive config stayed on the outer edge of the throughput-vs-latency curve at every point in the sweep, while fixed-length configs each dominated only at their tuned concurrency.

To enable it, you add two fields to your existing --speculative-config:

vllm serve deepseek-ai/DeepSeek-V4-Pro-0813 \
  --tokenizer-mode deepseek_v4 --trust-remote-code \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --kv-cache-dtype fp8 --max-model-len 16384 --max-num-seqs 256 \
  --speculative-config '{
    "method": "dspark",
    "attention_backend": "FLASH_ATTN",
    "num_speculative_tokens": 7,
    "draft_sample_method": "probabilistic",
    "enable_adaptive_verification": true
  }'

What it's good for, and where it falls short

Adaptive verification is a strong fit for any production serving scenario where traffic is bursty or concurrency varies across the day. Using the already-trained DSpark checkpoints via vLLM is an accessible path that does not require the full training infrastructure. The draft defaults to the target checkpoint, so no separate model download is needed.

There are real limitations to know about before deploying:

• Hardware constraint: Full varlen decode graphs require AttentionCGSupport.ALWAYS, which currently only the DSV4 sparse-MLA and sparse-SWA backends on SM100 (B300) report. On other hardware, adaptive verification is rejected at startup.
• No eager mode: --enforce-eager is not supported because step costs are profiled from captured graphs.
• No LoRA or pipeline parallelism support currently.
• No output logprobs: verification compacts logits after the forward pass, so per-token log probabilities are unavailable when adaptive verification is on.
• More backends coming: Flash Attention and DSV4 attention are supported now; other backends are in bring-up.

The bigger picture

DSpark demonstrates that the path to real-world serving efficiency lies in complete system co-design, and challenges the assumption that draft models must be purely autoregressive or purely parallel. The adaptive verification work in vLLM takes that a step further: the insight is that the right verification budget is not a deployment-time constant but a per-step variable that should track both the model's confidence and the GPU's current utilization.

By introducing a semi-autoregressive generation architecture, DSpark significantly extends the acceptance length of draft tokens, and the hardware-aware confidence-scheduled verification mechanism achieves a dual breakthrough in both low latency and high throughput for online systems. Adaptive verification in vLLM is the piece that makes that scheduling practical without requiring operators to profile and re-tune for every traffic pattern they encounter.

The feature landed in PR #47808 and is now merged into vLLM main behind the enable_adaptive_verification flag. It is free to use as part of vLLM's open-source release. More backends and model support are actively in development, with contributions from Lucas Wilkinson at Red Hat and Benjamin Chislett at NVIDIA.