LightSeek's TokenSpeed Hits 580 Tokens per Second on Qwen3.5's 400B Model

TokenSpeed, the open-source inference engine from the LightSeek Foundation, just set a new throughput record: 580 tokens per second on Alibaba's Qwen3.5-397B-A17B model, running on NVIDIA Blackwell GPUs. That's 580 tokens per second from a nearly 400-billion-parameter model -- a number that would have seemed implausible even a year ago. The result is a joint effort between the Qwen inference team, the LightSeek Foundation, NVIDIA, and the Mooncake team, with special contributions from Tri Dao on FlashAttention-4 (FA4).

Why Qwen3.5 is hard to serve fast

Before getting into the engine, it helps to understand what makes Qwen3.5 unusual. Qwen3.5 adopts a hybrid attention mechanism that interleaves standard full attention layers with linear attention layers based on the Gated Delta Network (GDN). Unlike traditional pure-Transformer architectures, this hybrid design maintains strong modeling capabilities while significantly reducing computational complexity for long-sequence inference.

GDN (Gated Delta Network) is essentially a smarter form of linear attention. Instead of maintaining an ever-growing attention map, each GDN layer keeps a fixed-size state matrix with dimensions proportional to the head dimension squared, independent of sequence length. New tokens update this state incrementally, and the output for each token is produced by querying the current state. The cost becomes O(n * d^2): linear in sequence length, quadratic only in the small, fixed head dimension. The model alternates between Gated DeltaNet layers (linear attention) and full attention layers in roughly a 3:1 ratio.

The catch: GDN layers carry a persistent recurrent state (like Mamba) that must be tracked, cached, and transferred alongside the conventional KV cache. That's a fundamentally harder serving problem than a standard Transformer, and it's where most inference engines fall short.

The engine built for agents, not chatbots

TokenSpeed is a speed-of-light LLM inference engine developed by the LightSeek Foundation, released as open source under the MIT license. The project targets agentic workloads and aims to deliver TensorRT-LLM-level performance with vLLM-level usability, combining a C++ control plane with a Python execution layer to keep CPU-side overhead minimal while preserving developer ergonomics.

Coding agents present unusually demanding inference workloads. Contexts routinely exceed 50K tokens, and conversations often span dozens of turns. Most public benchmarks do not fully capture this behavior. Because generation speed is crucial to the user experience for agents, the objective is to maximize per-GPU TPM (tokens per minute) while maintaining a per-user TPS floor -- typically 70 TPS, and sometimes 200 TPS or higher.

The 580 tps number, in context

The teams behind TokenSpeed and Alibaba's Qwen line posted a number to the PyTorch blog: 580 tokens per second, single user, serving the 397-billion-parameter Qwen3.5 on one NVIDIA Blackwell node. They're calling it a speed record for agentic workloads. The 580 figure is peak, measured at batch size one (one user, zero concurrency) on a B200 in TP8 configuration, with the model quantized to NVFP4, NVIDIA's 4-bit format.

All four parallelism configurations -- TP4, TP4EP4, TP8, and TP8EP8 -- sustain 500+ tok/s at batch size 1, with TP8 achieving a peak of ~580 tok/s. At concurrent=16, the TP4 family scales to ~2K tok/min/GPU system throughput while the TP8 family reaches ~1K tok/min/GPU.

Long-context performance holds up surprisingly well. On the NIAH (Needle-in-a-Haystack) benchmark, decode throughput remains at ~530 tok/s/user within 128K, ~495 at 256K, and ~445 at 1M tokens, giving an end-to-end degradation of only ~16% from 128K to 1M.

How they actually got there

This extreme performance is driven by systematic elimination of memory copies, advanced kernel fusions, and fully overlapped CPU-GPU execution -- keeping the GPU saturated at all times. Here are the key innovations:

• Hybrid prefix caching for GDN/Mamba state. Standard prefix caching reuses KV pages from previous turns. But for GDN layers, you also need the recurrent state at the exact prefix boundary. TokenSpeed attaches a MambaSlot checkpoint to the same radix-tree node as the KV prefix, enabling copy-on-write reuse of recurrent state across requests. The multi-turn agentic workload achieves an average KV cache hit rate exceeding 90%, significantly reducing prefill overhead.
• Move pointers, not data. In speculative decoding (where a draft model proposes tokens and the main model verifies them), the old approach copied the accepted Mamba state back into the working slot after every verification step -- a full tensor copy across all layers. TokenSpeed instead extends the state buffer with a draft region and uses a lightweight index table (current_input_indices) to track which physical row holds the canonical state. Post-verify bookkeeping becomes an O(1) integer write, not an O(L*D) tensor copy.
• Kernel fusion everywhere. Five separate HBM reads/writes for QK-RMSNorm + RoPE + gate split are collapsed into a single Triton kernel where all intermediate values stay in registers. The MoE shared expert gating path (5 kernels) is similarly fused into one. The AllReduce + residual + RMSNorm per decoder layer is merged from three kernel launches into one.
• CUDA graph capture of the full decode loop. The target model, sampler, and draft model are all captured into a single CUDA graph. Once captured, thousands of GPU kernels replay with one launch, eliminating per-kernel dispatch overhead entirely.
• Asynchronous everything. H2D transfers use pinned memory with non-blocking copies. The transfer system polls pinned-host counters instead of calling synchronize(). CPU-side index arithmetic (slot mappings, draft-token layouts) is annotated with torch.compile, letting Inductor fuse 10-14 individual launches into one or two elementwise kernels.

PD disaggregation for hybrid models

One of the more subtle engineering achievements is how TokenSpeed handles prefill-decode (PD) disaggregation -- the practice of running the prefill phase (processing the input prompt) on separate hardware from the decode phase (generating tokens). For a hybrid model, this means shipping both KV cache pages and Mamba recurrent states over the network between nodes.

TokenSpeed solves this with a unified step counter that ticks once after every layer's forward pass, regardless of whether it's an attention or GDN layer. The transfer thread watches this counter and ships whichever data belongs to each layer window -- KV pages for full-attention layers, state slots for GDN layers. A three-phase handshake ensures the decode node never begins generation with incomplete state, and the first output token is bundled with the final state transfer as a logically atomic unit.

What it's good at -- and where it falls short

TokenSpeed is purpose-built for a specific workload profile:

• Multi-turn agent loops with long shared context (tool call histories, code files)
• Single-user or low-concurrency deployments where per-user latency is the primary constraint
• Qwen3.5 and other hybrid attention models (GDN + MoE) on NVIDIA Blackwell
• Workloads where speculative decoding acceptance rates are high (long repeated prefixes)

The limitations are real. The peak 580 tps figure is measured at batch size one on a B200 in TP8 configuration with NVFP4 quantization -- a configuration that requires 8 high-end Blackwell GPUs. Performance on non-speculative or multimodal tasks remains unknown. TokenSpeed is currently under heavy development and is intended to showcase the new runtime design and technical direction.

The bigger picture

Inference efficiency has quietly become one of the most consequential bottlenecks in AI deployment. As agentic coding systems such as Claude Code, Codex, and Cursor scale from developer tools to infrastructure powering software development at large, the underlying inference engines serving those requests are under increasing strain.

What's notable about this result is that it wasn't achieved by throwing more hardware at the problem. It was achieved by treating every unnecessary memory copy, every CPU-GPU sync, and every un-fused kernel as a bug to be fixed. The team benchmarked their design against TensorRT-LLM -- the current state of the art on NVIDIA Blackwell -- and diverged from its approach wherever they believe better trade-offs exist for agentic workloads. On NVIDIA B200, TokenSpeed outperforms TensorRT-LLM by ~9% in min-latency and ~11% in throughput at 100 TPS/user on Kimi K2.5.

The field is also converging on a new assumption: that hybrid architectures mixing linear and full attention are the right bet for long-context agentic workloads. The Gated DeltaNet hybrid from Qwen3.5 offers a new direction. The benchmark landscape has shifted to match agentic workloads, and all four major recent releases target agentic tasks. Inference engines that can't handle recurrent state alongside KV cache will be left behind.

TokenSpeed is available now under the MIT license via the LightSeek GitHub repository. A Docker image (lightseekorg/tokenspeed-runner:latest) and benchmark reproduction scripts are available. FA4 support for Qwen3.5 on Blackwell is still in active development and will land in an upcoming release. Upcoming additions include model coverage for Qwen 3.6, DeepSeek V4, and MiniMax M2.7, along with Hopper and AMD MI350 optimizations.