Skip to main content
Back

DDTree Squeezes 8.22x Speed From DFlash Without Retraining

DDTree turns one block diffusion drafter pass into an optimal draft tree, lifting DFlash speedups from around 5x to over 7x across every tested benchmark.

Pro

  • DDTree builds a draft tree from a single DFlash block-diffusion forward pass, verified in one target-model call.
  • Best-first heap search recovers the top-B highest-probability prefixes in O(B log B), provably optimal for the surrogate objective.
  • Improves all 60 dataset-model-temperature settings tested, pushing Qwen3 speedups from around 5x to over 7x.
  • Peak result: 8.22x speedup on HumanEval with Qwen3-Coder-30B-A3B-Instruct at temperature 0.0.
  • Node budget sweet spot lands near 256-512; larger trees hurt due to verifier overhead.
  • Requires PyTorch SDPA for tree attention since FlashAttention-2 lacks the required mask pattern.

Speculative decoding has become the default trick for cutting large language model latency without touching output quality. A small drafter proposes several future tokens, and the target model verifies them in parallel, committing only tokens it would have generated anyway. The faster and more accurate the drafter, the bigger the speedup. A new method called DDTree squeezes noticeably more speed out of one of the strongest drafters available today by fixing an inefficiency almost nobody was exploiting.

The bottleneck hiding inside DFlash

DFlash is currently one of the leading drafters for speculative decoding. Rather than generate tokens one at a time like an autoregressive draft model, it uses a block-diffusion drafter that produces an entire block of candidate tokens in a single forward pass, turning sequential drafting into block-parallel GPU work. DFlash has been reported to deliver up to 2.5x higher speedup than the previous state of the art, EAGLE-3.

The DDTree authors zero in on a specific waste: a single block diffusion forward pass produces a full probability distribution over tokens at every future position, but vanilla DFlash collapses those distributions into a single drafted trajectory and hands that one path to the verifier. Every other plausible continuation the model already computed gets discarded.

From one path to a tree, cheaply