--- title: "EAGLE 3.1 Fixes a Silent Bug Making AI Inference 2x Faster in Production" subtitle: "EAGLE 3.1 fixes a newly discovered 'attention drift' flaw in speculative decoding, delivering up to 2x faster long-context throughput natively in vLLM" url: "https://alphasignal.ai/news/eagle-3-1-fixes-a-silent-bug-making-ai-inference-2x-faster-in-production" author: "AlphaSignal Newsroom" author_url: "https://alphasignal.ai/editorial-team" company: "vLLM" company_url: "https://alphasignal.ai/company/vllm_project" published: "2026-05-26T23:45:33.000Z" updated: "2026-06-08T19:35:22.551Z" type: "News" topics: - "Infra" - "Open Source" subtopics: - "Inference Optimization" - "Model Serving" upvotes: 344 read_time: "6 min" source_url: "https://vllm.ai/blog/2026-05-26-eagle-3-1" comments: 0 --- # EAGLE 3.1 Fixes a Silent Bug Making AI Inference 2x Faster in Production > EAGLE 3.1 fixes a newly discovered 'attention drift' flaw in speculative decoding, delivering up to 2x faster long-context throughput natively in vLLM Speculative decoding is one of the most impactful inference tricks in production LLM serving today. The idea is simple: a small, cheap draft model proposes several tokens at once, and the large target model verifies them all in a single forward pass. If the guesses are good, you get multiple tokens for the price of one. EAGLE achieves 2-3x speedups over standard autoregressive decoding and is widely deployed in production inference frameworks including vLLM, SGLang, and TensorRT-LLM. But there has been a quiet, persistent problem lurking in deeper speculation chains -- and the EAGLE team just found it, named it, and fixed it. The EAGLE series -- including EAGLE 1, EAGLE 2, and EAGLE 3 -- has become one of the most widely adopted families of speculative decoding algorithms across both research and production systems. Now, the EAGLE team, vLLM team, and TorchSpec team have jointly introduced [EAGLE 3.1](https://vllm.ai/blog/2026-05-26-eagle-3-1) -- a major step forward in speculative decoding robustness, efficiency, and deployability. ## The Bug That Nobody Had a Name For While speculative decoding performs well in controlled settings, performance often degrades under different chat templates, long-context inputs, or out-of-distribution system prompts. The EAGLE team traced this fragility to a phenomenon they call **attention drift** -- as speculation depth increases, the drafter gradually shifts attention away from sink tokens and toward its own generated tokens. Sink tokens are special tokens (like the beginning-of-sequence token) that transformers rely on as stable "anchors" for attention. When the drafter stops attending to them and starts over-attending to its own recent outputs, it loses its grounding in the original prompt -- and its token proposals become increasingly unreliable the further out it speculates. The team traced this to the unnormalized residual path between chain steps: the drafter's hidden state magnitude grows monotonically with chain depth, exhibiting dynamics consistent with additional pre-norm transformer layers stacked on the target rather than as a standalone autoregressive predictor. In other words, the deeper the speculation chain, the more the drafter's internal activations blow up in magnitude -- and the worse it gets at its job. This phenomenon was observed across both EAGLE3 drafters and MTP heads, suggesting drift is a property of drafter designs broadly -- not just an EAGLE quirk. The team [published a paper](https://arxiv.org/abs/2605.09992) formally characterizing the phenomenon. ## Two Small Changes, Big Impact The fix is elegant. EAGLE 3.1 introduces two targeted architectural changes to the draft model: - **FC normalization:** An RMSNorm (a lightweight normalization operation) is applied to each target hidden state before it is fed into the FC projection layer, preventing any single layer's activations from dominating the drafter's input. - **Post-norm hidden-state feedback:** The normalized hidden states -- rather than the raw ones -- are fed back into the next speculation step, keeping magnitude stable across the chain. ![Architecture diagram comparing EAGLE 3 pre-norm vs EAGLE 3.1 post-norm hidden-state feedback](https://vllm.ai/blog-assets/figures/2026-05-26-eagle-3-1/pre-norm-vs-post-norm.png) Intuitively, the post-norm design makes the drafter behave more like a true standalone autoregressive predictor at each step, rather than acting like extra layers bolted onto the target model. The result is a drafter that stays calibrated even at deeper speculation depths and across wildly different prompt formats. ## What the Numbers Look Like The gains are most pronounced in long-context workloads, which is exactly where EAGLE 3 was struggling most. In long-context workloads, EAGLE 3.1 achieves up to 2x longer acceptance length compared with EAGLE 3. Acceptance length is the key metric here -- it measures the average number of tokens the target model accepts per forward pass. Longer acceptance length means more tokens per GPU cycle, which translates directly to lower latency per user. The team benchmarked the Kimi K2.6 EAGLE 3.1 draft model on Kimi-K2.6-NVFP4 with vLLM (4-way tensor parallel on GB200 hardware) against a no-speculation baseline on the SPEED-Bench coding dataset. EAGLE 3.1 delivers 2.03x higher per-user output throughput at concurrency 1, and the speedup stays meaningful as concurrency scales -- 1.71x at C=4, 1.66x at C=16. ![Bar chart comparing per-user output throughput for Kimi-K2.6-NVFP4 baseline vs EAGLE 3.1 at concurrency levels 1, 4, and 16](https://vllm.ai/blog-assets/figures/2026-05-26-eagle-3-1/tpot_baseline_vs_eagle31.png) Beyond raw speed, EAGLE 3.1 also brings: - Stronger robustness to different chat templates and system prompts - Better generalization from training-time to inference-time conditions - More stable acceptance length across diverse serving environments ## How It Was Built TorchSpec lowers training overhead and simplifies experimentation workflows to accelerate iteration for next-generation speculative decoding research. Based on TorchSpec and vLLM, the team trained and open-sourced an EAGLE 3.1 draft model for Kimi K2.6, available on HuggingFace. The [Kimi K2.6 EAGLE 3.1 draft model](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3.1-mla) serves as a reference implementation showing the full pipeline from TorchSpec training to vLLM serving. NVIDIA provided GPU support for large-scale training, benchmarking, and inference validation. ## Dropping It Into vLLM EAGLE 3.1 lands in vLLM as a config-driven extension of the existing EAGLE 3 implementation, with full backward compatibility with existing EAGLE 3 checkpoints preserved. That means upgrading to a 3.1 draft model in production requires no code changes -- just a new model path in your `speculative-config`. The support has already been merged into vLLM's main branch and will ship in the upcoming **v0.22.0** release (also available via nightly builds now). Here is what serving Kimi K2.6 with EAGLE 3.1 looks like: ``` vllm serve nvidia/Kimi-K2.6-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 4 \ --speculative-config '{"model":"lightseekorg/kimi-k2.6-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}' \ --language-model-only ``` The method flag stays `"eagle3"` -- the 3.1 architecture is detected automatically from the checkpoint. This is a zero-friction upgrade for anyone already running EAGLE 3 in vLLM today. ## Why This Matters Beyond EAGLE The attention drift phenomenon was observed across both EAGLE3 drafters and MTP heads (the multi-token prediction heads used in models like DeepSeek), and the team traced it to the unnormalized residual path between chain steps. This suggests the fix is not EAGLE-specific -- any speculative decoding system that chains draft steps through a residual path may be silently suffering from the same instability. The architectural insight from this work could inform how future draft models are designed across the entire field. For teams running long-context inference workloads -- coding assistants, document analysis, agentic pipelines -- EAGLE 3.1 is a direct answer to the reliability gap that has made speculative decoding feel fragile in production. The combination of a named root cause, a clean architectural fix, open-source weights, and native vLLM support makes this unusually actionable. ## Sources - [X post by @vllm_project](https://x.com/vllm_project/status/2059420705834619104) - [EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec](https://vllm.ai/blog/2026-05-26-eagle-3-1) - [Attention Drift: What Autoregressive Speculative Decoding Models Learn](https://arxiv.org/abs/2605.09992) - [P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM](https://vllm.ai/blog/2026-03-13-p-eagle) - [Eagle3 - Speculators Docs - vLLM Documentation](https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/eagle3/) - [Meet EAGLE 3.1: The Speculative Decoding Algorithm That Fixes Attention Drift in LLM Inference](https://www.marktechpost.com/2026/05/27/meet-eagle-3-1-the-speculative-decoding-algorithm-that-fixes-attention-drift-in-llm-inference/)