DeepSeek's DSpark Makes DeepSeek-V4 Generation 85% Faster at Any Scale

Speculative decoding has always had a dirty secret: it works brilliantly at low concurrency, and quietly falls apart as your server fills up. SGLang now integrates DSpark, a new speculative decoding framework from DeepSeek, that attacks this problem head-on with confidence-driven, variable-length verification. The result is the best throughput/latency tradeoff across the full concurrency range, from a single user to 256 concurrent requests.

Why standard speculative decoding breaks under load

The standard speculative decoding loop works like this: a small, cheap draft model proposes a block of K tokens, and the large target model verifies all of them in one forward pass. At batch size B, that means verifying B * K tokens every step. Parallel drafters efficiently propose long token sequences in a single forward pass, but suffer from rapid acceptance decay due to a lack of inter-token dependencies. Indiscriminately verifying these extended blocks wastes critical batch capacity on tokens with high rejection risks, severely degrading throughput in high-concurrency serving systems.

Speculation spends spare compute to skip memory reads, so its benefit collapses as concurrency rises and the GPU becomes compute-bound. At batch size one you have abundant idle compute and speculation is a clear win. On a server saturated with concurrent requests, the extra verification work competes for compute that is already busy, and the speedup trends toward zero or negative. Most frameworks just pick a fixed verification budget and hope for the best.

What DSpark actually does differently

The full name is Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. The core idea is simple: draft more than one token, verify intelligently, and avoid spending expensive target-model compute on tokens that are unlikely to survive. DSpark achieves this through two interlocking mechanisms:

• Semi-autoregressive block drafter: DSpark utilizes a semi-autoregressive architecture, coupling a parallel backbone with a lightweight sequential module, to introduce intra-block dependency modeling and mitigate suffix decay. The sequential head (a Markov chain or small RNN) conditions each draft token on the previous one, keeping acceptance rates high across the whole block without the cost of full autoregression.
• Confidence-scheduled verification: DSpark employs confidence-scheduled verification, dynamically tailoring the verification length for each request based on estimated prefix survival probabilities and engine-specific throughput profiles. Instead of verifying all K draft tokens, it only verifies as many as the confidence score says are worth checking.

Rejection sampling accepts the longest valid prefix and appends one bonus token. Because the rule preserves the target distribution exactly, there is no quality loss. DSpark keeps this guarantee. The output is lossless by construction.

The SGLang integration: where the engineering gets interesting

Getting the DSpark algorithm to actually run fast in a production server required significant new engineering inside SGLang. The core challenge: per-request variable verify lengths don't fit a fixed-shape CUDA graph, and naively padding every request to the full block width just undoes the savings from trimming.

The SGLang team solved this with a ragged verify path. Instead of padding, requests are front-packed into a compact buffer and the graph is keyed on the total token count, rounded up to the nearest pre-captured tier. When confidence-based trimming cuts the budget, the total drops to a smaller tier and the engine replays a genuinely cheaper graph with fewer attention and MLP rows. Under data-parallel attention, all ranks step down together.

The other major addition is Zero-Overhead Scheduling (ZOS). SGLang's spec-v2 runtime already overlaps the next step's scheduling with the current forward on separate streams, and DSpark joins as a first-class worker. The confidence relay uses the same async channel, read two steps back. The result: no per-step bubble in the decode loop, running about 1.5x tighter than with the scheduler off.

Additional engineering highlights:

• Fused Triton kernels for the compact scatter, SWA page-index, verify-length top-k schedule, and ragged-window packing. In one example profile, overhead outside the target verify shrank by 1.7 ms against a 7.3 ms verify.
• Additive SPS cost table: an offline-profiled step-time model (T(bs, K) = bias + alpha(bs) + theta(M)) that the scheduler reads online to decide the optimal verify budget per step.
• Three verify modes: static (full block, baseline), compact (production path, trimmed per request), and cap-accept (full verify but commits only up to the window, used to measure the acceptance ceiling without censoring it).
• Per-request differentiation on mixed traffic: a batch with gsm8k (high acceptance), arena-hard (mid), and poetry (low) requests gets different verify windows per request. Window sizes were 5.24, 3.78, and 2.91 tokens respectively, while utilization against the acceptance ceiling stayed between 0.88 and 0.97.

The numbers

DSpark makes per-user DeepSeek-V4 generation 60-85% faster on V4-Flash and 57-78% faster on V4-Pro, without changing the model's weights or its output quality. On the SGLang benchmark specifically:

• DSpark delivers the best throughput/latency tradeoff across batch sizes 1 to 256, ahead of both MTP (the EAGLE-style baseline) and non-speculative decoding.
• At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget.
• DeepSeek-V4-Pro reaches 383.7 tok/s at batch size 1 on B300 (TP8) with an accept length of ~5.
• On Qwen3-4B, DSpark improved the macro-average accepted token length by 30.9% over Eagle3, the prior state-of-the-art autoregressive speculative drafter.

The dynamic scheduling win is primarily a high-batch effect. At batch size 1, trimming saves little because the verify cost is low anyway. As concurrency grows and throughput starts to plateau, trimming shortens each step and compact mode pulls ahead. The gap is larger on lower-acceptance workloads, exactly as the cost model predicts.

How to run it today

Using the already-trained DSpark checkpoints via SGLang is a separate and more accessible path that does not require the full training infrastructure. DSpark is available now in SGLang with support for Qwen3 and DeepSeek-V4 (both Flash and Pro). The model checkpoints (DeepSeek-V4-Pro-DSpark and DeepSeek-V4-Flash-DSpark) are the same weights as the base models with a draft module attached.

To launch DeepSeek-V4-Pro with DSpark on B300 (TP8):

SGLANG_RAGGED_VERIFY_MODE=compact python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-DSpark \
  --speculative-algorithm DSPARK \
  --tp 8 \
  --moe-runner-backend flashinfer_mxfp4 \
  --mem-fraction-static 0.82 --cuda-graph-max-bs 192 --chunked-prefill-size 4096 \
  --trust-remote-code --host 0.0.0.0 --port 30000

The key environment variable is SGLANG_RAGGED_VERIFY_MODE=compact, which activates the production path with dynamic per-request trimming. You can also use static for the full-block baseline or cap-accept to measure the acceptance ceiling on your own traffic.

The bigger picture

DSpark demonstrates that the path to unlocking real-world serving efficiency does not lie in isolated algorithmic or architectural optimization, but in a complete system co-design. The algorithm, the confidence head, the CUDA graph path, the scheduler, and the fused kernels all had to be built together. This signals that speculative decoding is transitioning from a nice-to-have acceleration add-on into a standard component of the large model inference stack.

For teams not running DeepSeek models, DeepSeek also released DeepSpec, an MIT-licensed codebase for training and evaluating the small draft models the technique relies on. Community testing confirmed that DSpark trains cleanly on Qwen3 and Gemma targets, meaning teams not using DeepSeek models at all can apply the method. The roadmap for SGLang includes a stronger adaptive cost model, broader model coverage, and productionized observability metrics like the block-accept estimator.