Z Lab's DFlash Hits 4.3x Throughput by Drafting Tokens in Parallel

Speculative decoding has been one of the most impactful inference tricks of the past two years: use a small, cheap draft model to guess several tokens ahead, then let the big target model verify them all in one shot. The catch is that most draft models, including the popular EAGLE series, still generate those guesses one token at a time. That sequential bottleneck caps how fast you can go. DFlash, developed by Z Lab and now integrated into SGLang with Modal's help, breaks that ceiling.

Using Modal and Z Lab's DFlash speculative decoding models with SGLang's newly default Spec V2 engine, you can achieve state-of-the-art latencies for LLM inference serving. The headline number: the jointly-released DFlash model for Qwen 3.5 397B-A17B achieves higher throughput than both the baseline model and native MTP speculation in all the settings benchmarked, hitting over 4.3x the throughput of baseline and 1.5x the throughput of MTP at concurrency 1 on the HumanEval coding dataset on 8xB200 GPUs.

The draft model bottleneck nobody talks about

GPUs offer massive compute, yet much of that power sits idle because autoregressive generation is inherently sequential: each token requires a full forward pass, reloading weights, and synchronizing memory at every step. Speculative decoding was supposed to fix this, but it only moved the problem. EAGLE is the state-of-the-art method for speculative decoding in LLM inference, but its autoregressive drafting creates a hidden bottleneck: the more tokens you speculate, the more sequential forward passes the drafter needs.

The same problem applies to native multi-token prediction (MTP) modules baked into models like DeepSeek-V4 and Gemma 4. Many speculative decoding methods still rely on sequential autoregression, but in the draft model instead of the target. The draft model generates draft tokens one-by-one, a poor fit for modern hardware and a limit on achievable speedup.

Two ideas that change the math

DFlash combines two distinct innovations that each independently improve performance, and compound when used together.

Block diffusion drafting. Z Lab developed DFlash, which uses a lightweight block diffusion draft model to generate an entire block of draft tokens in parallel, just the way GPUs and TPUs like. Block diffusion (a term borrowed from diffusion language models) means the draft model fills in a whole sequence of masked token positions simultaneously in a single forward pass, rather than left-to-right one at a time. With DFlash, K tokens cost the same as 1 token in the draft phase, so higher K values are more efficient with DFlash than with EAGLE-3.

KV injection. Fast drafting only helps if the drafted tokens get accepted by the target model. The key insight of DFlash is that the target LLM knows the context best. DFlash extracts hidden representations of the context tokens from the target model and injects them directly into the draft model's KV cache. KV cache here refers to the stored key-value attention tensors that represent context. By injecting the target model's own context representations into every layer of the draft model, the drafter stays strongly conditioned on the big model's understanding throughout generation. This scales better with increased draft depth and allows the draft model to skip modeling the full context from scratch and focus purely on predicting the next block of tokens.

The ablation numbers make the contribution of each technique clear:

• Diffusion only (no KV injection): DFlash still beats EAGLE-3 on GSM8K (2.9x vs 2.1x speedup) despite lower acceptance length, purely from faster drafting
• KV injection only (autoregressive mode): Acceptance length jumps to 4.8 on GSM8K vs EAGLE-3's 4.2, showing the quality boost from target-model conditioning
• Both combined: 3.3x speedup on GSM8K vs EAGLE-3's 2.1x, with comparable acceptance length

The Spec V2 engine: squeezing out the last 33%

Even with a faster draft model, there is overhead from the CPU-side scheduler coordinating with the GPU. Every time the scheduler has to wait for the GPU to finish before it can prepare the next batch, you lose time. The key goal of the V2 engine is to reduce points of host-device synchronization, which kill inference performance no matter how fast the GPU is or how good the kernels are. The solution is called the overlap scheduler.

The overlap scheduler exploits two specific opportunities:

1. Post-batch cleanup on the CPU (stop token detection, request metadata updates) can run while the GPU is already working on the next batch
2. KV cache memory allocation for batch N can overlap with GPU execution of batch N-1

Under V2 with these optimizations, performance improved by over 33%, from ~11.4 ktok/s to ~15.3 ktok/s, when running Qwen 3-8B on a single B200 at concurrency 32. Spec V2 is now the default speculative decoding engine in SGLang.

How it was built

The Qwen3-Coder-30B-A3B-DFlash model was trained on 289K samples, composed of code from the Nvidia Nemotron Post-Training Dataset. Despite being trained on significantly less data, DFlash already outperforms EAGLE-3 in inference acceleration. In comparison, the EAGLE3 model for the same target was trained on the open-perfect-blend dataset with 1.4M samples, nearly 5x more data than DFlash. The DFlash architecture's strong conditioning via KV injection means the draft model needs less training data to achieve high acceptance rates.

The implementation in SGLang required solving a subtle engineering problem: KV injection ties state between the draft and target models, which conflicts with SGLang's radix cache (a prefix-sharing mechanism). The team uses immediate materialization: running the draft KV projection ahead of the rest of the draft forward pass, and added a layer-batched linear projection and a fused Triton kernel for the norm+RoPE post-processing to keep it fast.

Already in production at scale

The real-world validation is already here. Xiaomi paired with TileRT_AI to push an optimized serving mode of their existing 1-trillion-parameter MoE model, claiming the first reported inference above 1,000 tokens per second on ordinary 8-GPU hardware through FP4 quantization and a block-level speculative decoding method called DFlash. For context, Cerebras achieved 969 TPS on Meta's Llama 3.1 405B using wafer-scale integration that cost hundreds of millions to develop. Xiaomi did it on a model 2.5x larger, on rentable cloud hardware.

What's available and how to use it

The DFlash drafter for Qwen 3.5 397B-A17B is available now on Hugging Face under three organizations: z-lab/Qwen3.5-397B-A17B-DFlash, modal-labs/Qwen3.5-397B-A17B-DFlash, and lmsys/Qwen3.5-397B-A17B-DFlash. NVIDIA's work is also bringing DFlash to vLLM, making it available to the broader serving community. You can also find additional draft models in Z Lab's DFlash collection on Hugging Face, including drafters for GPT-OSS 120B and Qwen3-Coder-30B.

To spin up a DFlash-accelerated server with SGLang:

export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path modal-labs/Qwen3.5-397B-A17B-DFlash \
  --speculative-dflash-block-size 8 \
  --speculative-draft-attention-backend fa4 \
  --attention-backend trtllm_mha \
  --tp-size 8

What this means for the field

DFlash challenges a core assumption that has shaped speculative decoding research: that the draft model's job is to autoregressively predict tokens as accurately as possible. The DFlash result shows that drafting speed matters as much as draft quality, and that a parallel drafter with lower acceptance length can still outperform a sequential drafter with higher acceptance length. Speculative decoding stopped being a research-paper optimization in 2026 and became the default inference acceleration layer of every serious LLM stack. DFlash is now the most capable option in that stack for supported models.

The practical use cases that benefit most are those with low concurrency and long outputs: coding agents, chain-of-thought reasoning, and multi-turn conversations. SGLang is particularly effective with DFlash on multi-turn agent workloads. SGLang's RadixAttention caches KV state across turns, so each subsequent turn in a conversation is a shorter prefill. DFlash's block diffusion draft reduces the per-turn decode latency, compounding the RadixAttention benefit. If you are running a high-concurrency batch inference job, the gains are smaller but still present across all benchmarks tested.

You can read the full SGLang integration code, deploy a server with the command above, or reach out to Z Lab if you want a custom DFlash drafter trained for your own model or data distribution.