SGLang's Low-Latency Mode Boosts DeepSeek-V4.1-Flash Speed by 1.56x on AMD

SGLang has added Low-Latency Mode, a plugin path for vendor-specific megakernels developed with NVIDIA and AMD. The mode targets interactive inference, where a single request leaves GPUs running small kernels and waiting on cross-GPU reductions. Large batches amortize those costs across thousands of tokens; low-concurrency agent workloads cannot.

Published benchmarks replay SemiAnalysis’s AgentX trace on DeepSeek-V4.1-Flash with DSpark speculative decoding, which proposes several tokens and verifies them together. All results compare the plugin with SGLang’s baseline on the same four-GPU system.

Six rows keep four GPUs waiting

In the NVIDIA batch-size-one profile, DSpark verifies only six token rows per step. Each GPU kernel therefore performs little work, while dependency chains and NVLink reductions consume much of the elapsed time. The workload is limited by synchronization and communication rather than arithmetic throughput.

After CUDA graph capture and warm-up, the NVIDIA plugin reduces the verification graph from 1,205 kernels to 1,189, a 1.3% decrease. Median decoder-layer latency falls from 96.8 µs to 68.4 µs, or 29.3%. Earlier dependency releases and overlapped communication account for most of the reduction.

NVIDIA overlaps the dependency chain

The sgl-nvidia-mega library relies on Programmatic Dependent Launch, a Hopper and Blackwell feature that lets a dependent kernel prepare barriers and load weights before its input tensor is ready. SGLang already used PDL; the plugin releases dependencies earlier, allowing more setup to overlap with active computation.

• Epilogue multicast: After the attention output projection, each thread block, or CTA, publishes completed tiles directly from registers to peer GPUs through NVLink multicast stores. The reduction kernel starts through PDL and waits for each required tile. This path avoids writing partial sums to high-bandwidth memory solely to transmit them.
• Routing without sorting: Each expert receives no more than eight token rows, so its work fits in one eight-row tile. A bitmap and population counts place the tiles without a scan or sort while preserving SGLang’s top-k routing choices.
• Single-kernel draft sampling: One launch per GPU handles all five proposals and keeps head weights in registers while coordinating token selection across GPUs. The profiled proposal span drops from 85.1 µs to 34.2 µs without changing SGLang’s speculative worker or acceptance logic.

AMD keeps the layer resident

The sgl-amd-mono library collapses most of each decoder layer into two persistent kernels. K1 performs the model’s mHC residual-state update, query and key-value projections, and key-value cache writes. K2 handles attention, output projections, the remaining mHC update, mixture-of-experts computation, and both all-reduces that combine partial tensors across GPUs.

HIP graphs remove CPU launch work, but each small kernel still costs 2 to 4 µs on the GPU. Compute units idle while one kernel drains and the next ramps up, and each layer pauses twice for all-reduce operations. Keeping most of the layer inside two kernels removes many of those transitions.

Within K2, each completed output-projection tile is pushed into every GPU’s memory. Each GPU sums the pieces in rank order while other workgroups continue producing tiles. Every cross-GPU wait also has a deadline shorter than SGLang’s watchdog timeout; a missed deadline stops the launch on every GPU instead of leaving the server hung.

Four stages keep megakernels outside core

SGLang’s plugin boundary gives specialized kernel libraries a defined path into the runtime, reducing the need to place model-specific and vendor-specific code in the core serving repository. The contract has four stages:

1. Register: Expose the plugin implementation to SGLang.
2. Guard: Check whether the model, hardware, tensor shapes, and step type are supported.
3. Build: Prepare the selected execution path and its required state.
4. Runtime: Execute the optimized path for eligible work.

All tensor-parallel ranks must agree that a plugin can handle the operation before entering its collective communication path. Unsupported shapes and step types fall back to SGLang’s general forward pass within the same request. This agreement prevents ranks from entering incompatible collectives, which could otherwise deadlock the server.

Where the mode fits today

Low-Latency Mode currently accelerates DeepSeek-V4.1-Flash with DSpark on NVIDIA GB300 and AMD MI355X or MI350X systems. The narrow support matrix makes the model, decoding method, accelerator, and expected concurrency the main deployment checks.

On a workload with 8,000 input tokens and 1,000 output tokens, the NVIDIA plugin improves per-user decoding speed by 12% to 38% across one to eight concurrent requests on four GB300 GPUs. The published results focus on low concurrency, where launch overhead and reduction stalls dominate, and do not establish a throughput advantage for large-batch serving.

SGLang’s roadmap lists megakernel or monokernel work for GLM-5.3, K3, MiniMax M3, and additional models on NVIDIA and AMD hardware. A stable plugin interface would let vendors distribute per-model kernels while applications retain SGLang’s shared scheduler, fallback paths, and serving APIs.