Skip to main content
Back

UC Berkeley Exposes How Speculative Decoding Speedups Collapse at Real Batch Sizes

A production-grade benchmark of speculative decoding in vLLM reveals that verification cost dominates, batch size erodes gains, and the gap to theoretical maximum is huge.

Pro

  • First systematic evaluation of speculative decoding inside vLLM, not a prototype at batch size 1.
  • Verification cost dominates end-to-end runtime, taking 42-95% of execution time across variants.
  • Speedup drops sharply with batch size: EAGLE on Llama3.1-8B falls from 1.73x to 1.21x from batch 1 to 128.
  • N-gram beats EAGLE on code editing when prompt-output BLEU overlap exceeds ~0.6.
  • Tree verification helps only at batch size 1 and hurts throughput at larger batches.
  • Oracle upper bound combining methods reaches 4.9x speedup, exposing large headroom; code at SpecDecode-Bench.

Speculative decoding has become the go-to trick for squeezing more tokens per second out of large language models. The idea is simple: use a cheap drafter to guess several tokens ahead, then have the big model verify them all in parallel. Almost every paper claiming impressive speedups ran experiments on toy prototypes with a batch size of one, a setup that looks nothing like real inference servers. A new study out of UC Berkeley finally puts the technique through its paces inside a production engine, and the results reshape a lot of common wisdom.

The authors ran the first systematic evaluation of speculative decoding inside vLLM, benchmarking n-gram, EAGLE, EAGLE-3, draft-model, and Multi-Token Prediction (MTP) variants across six datasets and model sizes from 8B up to 106B. Every SD variant outperforms the no-SD baseline, but the speedup decreases as the batch size increases. That last part is where the illusion starts to break.

The batch size cliff

At batch size 1, speculative decoding looks amazing. Crank the concurrency up to what a real server actually runs, and the gains melt. On Llama3.1-8B with GSM8K, EAGLE's speedup over the non-SD baseline drops from 1.73x to 1.21x as the batch size goes from 1 to 128. The effect worsens on bigger models: Llama3-70B on ShareGPT loses 14% of its speedup going from batch 1 to 32, versus 4.3% on the 8B.

Why? Small batches leave GPUs idle and memory-bound, so there's plenty of spare compute to verify tokens that end up rejected. As batches grow, the system becomes compute-bound and every wasted verification hurts. Verification generally takes the largest share of execution time, ranging from 42% to 95% across all methods, increasing with model size and batch size. Drafting itself is cheap: n-gram drafting is under 2% of runtime, while EAGLE-style heads add 12-20% at batch size 1 and only 3-7% at batch size 512.