ReplaySSM cuts recurrent-state traffic during decoding

State-space models (SSMs) and recurrent linear-attention layers can decode long sequences without an attention key-value (KV) cache that grows with every token. Each layer compresses prior tokens into a fixed-size recurrent state, keeping per-token storage and computation constant as context length grows. Hybrid models built with Mamba-2 or Gated DeltaNet still face a costly operation: every decoding step reads and rewrites that state in high-bandwidth memory (HBM), which can make recurrent layers the latency bottleneck.

Researchers from Princeton’s Dao AI Lab and NVIDIA propose ReplaySSM in a new arXiv paper. The method caches a short window of layer inputs in a ring buffer, reconstructs the current state when producing outputs, and commits buffered inputs to the recurrent state periodically. The authors report integrations with NVIDIA TensorRT-LLM, vLLM, and FlashInfer, plus production use in GLM-5.3-Flash and Kimi-K3.

Fixed state, expensive traffic

Conventional recurrent decoding eagerly commits every token to state. A fused kernel loads the previous state, applies the new input, computes the layer output, and writes the updated state to HBM. The state stays constant in size, but transferring it across many layers and concurrent sequences consumes substantial memory bandwidth.

• State updates are memory-bound. Each step moves the full state while performing relatively little arithmetic. In the hybrid architectures studied, SSM layers outnumber attention layers by roughly five to one, so their aggregate traffic dominates at many practical context lengths.
• Speculative rollback is expensive. Speculative decoding proposes several tokens and verifies them together. When verification rejects a suffix, an eagerly updated SSM state already includes those tokens, requiring restoration of an earlier state or recomputation.