LMSYS Doubles Muse Spark Open Speed to 1,109 Tokens per Second

LMSYS has more than doubled Muse Spark Open’s reported batch-one decode throughput on eight NVIDIA GB300 GPUs. Its technical report describes a serving stack that fuses transformer operations into long-running CUDA megakernels, raising throughput from 517 to 1,109 generated tokens per second.

The serving stack keeps one thread block active on each participating streaming multiprocessor, or SM. Warps inside those blocks coordinate matrix math, weight movement, attention, and cross-GPU transfers. Moving that scheduling into GPU code allows independent hardware engines to overlap work across boundaries that SGLang handles as separate kernel launches.

The decode-step measurements use a GSM8K request with tensor parallelism across all eight GPUs. Tensor parallelism splits each model layer’s computation among the devices, which must exchange partial results over NVLink.

Launch boundaries leave hardware idle

A conventional CUDA kernel implements one operation, such as matrix multiplication or attention, across a grid of thread blocks. During Muse Spark Open decoding, SGLang chains roughly 1,450 such launches into each step. Dependencies between them can leave tensor cores, load units, the Tensor Memory Accelerator bulk-copy engine, and NVLink waiting for the next operation.

Each fused segment begins with attention in one layer and continues through the following layer’s query, key, and value projection and key-value cache write. The cache stores attention keys and values from earlier tokens. Device-side coordination keeps data moving through this operation chain while reducing the full step to 97 launches.

A 12-warp pipeline takes over scheduling

Each of 144 SMs runs a thread block containing 12 warps, or 384 threads. Blocks launch in clusters of four and use distributed shared memory, which lets blocks in the same cluster exchange data with lower latency than global memory.

Warps receive fixed roles within each phase. Some stream weights, others issue tensor-core work, and another group handles the attention epilogue. A dedicated warp warms the instruction cache. The resulting software pipeline resembles cooperative multithreading, with warps acting as specialized workers and device-side counters coordinating handoffs.

Four mechanisms keep work moving

• Weight prefetching: Model weights do not depend on the preceding layer’s output, so warp 0 can fetch weights for the next operation while the current operation runs. One quarter of the attention output-projection weights enters shared memory during attention execution, with additional data staged in L2 cache.
• Speculative expert prefetching: Mixture-of-experts layers route each token through selected expert subnetworks. Because routing often repeats across consecutive decode steps, warp 3 preloads the 14 most-used experts from the previous step into L2. The transfer covers about 50 MB per GPU and saves roughly 0.2 ms on a reported 5.3 ms step.
• Instruction-cache warming: The compiled kernels occupy 215 to 314 KB, exceeding the SM’s approximately 128 KB instruction cache. At each launch, a warp touches one word from every 128-byte cache line, pulling code into L2 before execution reaches it.
• Fence-free cross-GPU synchronization: Receivers initialize their buffers with an empty sentinel and poll the payload directly. This removes the write, memory-fence, and flag sequence, which costs 3.6 to 4.0 µs when bulk loads are active. For bf16 gathers, the sentinel is an all-bits-set NaN value that rounding cannot produce.

Draft and verify share one graph

Muse Spark Open uses DFlash V2 as its speculative-decoding draft model. The smaller draft proposes 15 upcoming tokens in one pass, and the target model evaluates a 16-token verification tree in parallel. Accepted candidates become output tokens; rejected branches are discarded.

Fusion reduces the draft model’s five layers to one launch apiece, with supporting operations bringing the total to eight launches. Drafting and verification also share one CUDA graph, a recorded sequence of GPU work that can be replayed with less launch overhead. These changes account for the reported 2.6x reduction in draft-step time.

Xiaomi’s MiMo report describes MiMo-V2.5-Pro exceeding 1,000 tokens per second while verifying eight tokens and accepting 6.3 on average. LMSYS verifies 16 and accepts 6.6, spending more target-model computation for a slightly longer accepted sequence. End-to-end comparisons also depend on the model, hardware, kernels, sampling configuration, and benchmark workload.

Batch one defines the current limit

The current implementation covers speculative decoding at batch size one, so servers that batch concurrent requests cannot use this path yet. Batch-one inference exposes synchronization gaps because the GPU has little unrelated work available to fill them, making it a demanding latency target.

The authors list larger batches on the roadmap and expect additional tokens in flight to provide more independent work for the warps. All published results use eight GB300 GPUs, and performance on other GPU generations remains unreported. Porting the design would require fresh tuning for cache capacity, instruction behavior, communication, and supported GPU features.

Eight thousand lines behind 5.9 ms

The GitHub repository contains about 8,000 lines of CUDA C++ and PTX, NVIDIA’s low-level GPU instruction language. The implementation exposes the model-specific fusion boundaries, cache management, communication routines, and synchronization needed to reproduce the benchmark.

The authors propose extending the design into a GPU-resident runtime where warps serve as workers, counters track dependencies, and a scheduler dispatches ready tasks dynamically. Achieving the current result already requires model-aware fusion, architecture-specific tuning, custom collectives, and scheduling logic encoded directly in GPU code.