Skip to main content
Back

Mamba Quietly Beats Transformers on Speed, Memory, and Long Sequences

A comprehensive review published in ACM TIST maps how selective state space models challenge Transformers with linear scaling and strong long-sequence performance.

Pro

  • Survey of Mamba consolidates the fast-moving literature on selective state space models as Transformer alternatives, published in ACM TIST.
  • Mamba combines HiPPO initialization, input-dependent selection, and hardware-aware parallel scan for 3x faster A100 computation.
  • Mamba-2 introduces Structured State-Space Duality linking SSMs to linear attention, running 2-8x faster than Mamba-1.
  • Concrete wins include Vim's 86.8% GPU memory savings and Jamba's 32x smaller KV cache versus LLaMA-2-7B.
  • Adaptations cover images, graphs, point clouds, speech, DNA, molecules, recommenders, and robotics via scanning tricks.
  • Known weaknesses: fails classic copying beyond state size, weak state-tracking, and inherently causal design limits non-autoregressive tasks.

Transformers still run the show, but a rival family of models has been quietly compiling wins across language, vision, biology, and robotics. A newly updated survey accepted by ACM Transactions on Intelligent Systems and Technology pulls together the sprawling literature on Mamba, an architecture built on selective state space models (SSMs) that scales near-linearly with sequence length. It covers architecture evolution, adaptation techniques for non-sequential data, real applications, and the surprisingly stubborn limitations that keep Mamba from being a drop-in Transformer replacement.

Why attention hit a wall

The complaint is familiar. Transformer architecture underpins most advanced models today, especially large language models with billions of parameters, but the self-attention mechanism carries a heavy tax. Its quadratic computation complexity makes inference slow and pushes long documents, hour-long videos, and genome-scale sequences out of reach.

Mamba draws on a much older tradition: classical state space models from control theory. The survey positions the architecture as a fusion of RNN-style recurrence and CNN-style convolution, delivering Transformer-grade modeling with linear scaling, fast parallel training, and fast recurrent inference at once.

Diagram comparing RNN, Transformer, and SSM computation patterns

The three tricks that make Mamba work

Classical SSMs are elegant but limited: their transition matrices are fixed regardless of input, which cripples them on tasks like selective copying. Mamba-1 rebuilds the SSM around three ideas.

  • HiPPO-based memory initialization. The hidden state matrix is initialized using HiPPO theory with a scaled Legendre measure, so the model starts out able to compress long histories without collapsing older tokens.
  • A selection mechanism. Instead of static matrices, Mamba makes the SSM's B, C, and step size Δ functions of the input. This gives the model something close to attention's content-awareness while keeping the recurrent form.