EAGLE-4 Triples Qwen3 Inference Speed by Adapting to Serving Load
A single adaptive drafter adjusts speculative decoding to batch size and hardware, posting up to 2.99x throughput on vLLM and 3x single-request speedups.
- EAGLE-4 ships as eagle-harness, a source-available drafter that plugs into SGLang, vLLM, and TokenSpeed.
- One drafter adapts tree depth and branching to batch size, hardware and workload at runtime.
- Measured 2.99x on vLLM batch-of-1 and 1.35x on SGLang 64-way, all preserving target distribution.
- Qwen3.8-27B is the reference target; draft weights are published on Hugging Face.
- Supports NVIDIA sm_80+ and AMD CDNA2 through CDNA4, with MI355X verified by serving tests.
- Commercial use capped at 20M USD annual revenue; competing inference services require a paid license.
The EAGLE team has released EAGLE-4, an adaptive speculative-decoding drafter, alongside a pip-installable package called eagle-harness. The drafter measures serving costs and adjusts its speculative tree for the current batch size, hardware, and workload, reducing the need to maintain separate configurations for interactive and high-concurrency inference.
Speculative decoding uses a smaller model to propose several tokens before the target model verifies them in parallel. Correct verification preserves the target model’s output distribution while reducing the number of sequential target-model calls. Performance depends on how quickly the drafter proposes useful candidates and how much work the target spends verifying them.
pip install eagle-harnessThe harness integrates with SGLang, vLLM, and the experimental TokenSpeed engine through their registration hooks, with no runtime patching. The reference pairing uses Qwen3.8-27B and a published EAGLE-4 checkpoint.
Runtime-sized speculation
EAGLE-4 profiles draft generation and target verification at the active batch size, then selects the depth and branching factor expected to produce the highest throughput. A top-k value of 1 creates a speculative chain, while larger values create a tree of candidate continuations.
The drafter combines several techniques intended to limit its own compute and memory costs:
- Sliding-window draft layer: Keeps a bounded window of recent hidden states instead of attending across the full sequence.
- Low-rank candidate head: Factors token prediction into smaller matrices, reducing the cost of proposing candidates.
- Append-only draft cache: Extends the draft KV cache between rounds instead of rebuilding it.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
