CMU's Mamba-3 Beats Transformers on Speed While Outperforming Rival SSMs
Mamba-3 redesigns state space models from the ground up for inference, beating Transformers on latency while matching or exceeding them on accuracy at the 1.5B scale.
ProTakeaways
- Mamba-3 released: New SSM from CMU, Princeton, Cartesia AI & Together AI, designed for inference efficiency over training speed.
- Beats Transformers on latency: At 1.5B scale, Mamba-3 SISO is faster than vLLM-optimized Llama-3.2-1B across all sequence lengths on a single H100.
- Three core innovations: Exponential-trapezoidal discretization, complex-valued state transitions (via RoPE trick), and MIMO (multi-input, multi-output) SSMs.
- MIMO accuracy gain: Mamba-3 MIMO hits 57.6% downstream accuracy at 1.5B, beating Llama-3.2-1B by +2.2pp and Gated DeltaNet by +1.8pp.
- Open-source under Apache 2.0: Code and custom kernels (Triton, TileLang, CuTe DSL) available at state-spaces/mamba.
- Limitation: Pure retrieval tasks still favor Transformers; team recommends hybrid Mamba-3 + attention architectures for retrieval-heavy workloads.
The Mamba team is back with a third generation of their state space model (SSM) architecture, and this time the design philosophy has flipped entirely. Where Mamba-2 was built to train fast, Mamba-3 is built to run fast. The result, published at ICLR 2026 by researchers from CMU, Princeton, Cartesia AI, and Together AI, is an open-source SSM that beats Transformers on prefill+decode latency across all tested sequence lengths at the 1.5B scale, while also outperforming Mamba-2 on downstream accuracy benchmarks.
Why Mamba-2 left performance on the table
To understand what Mamba-3 fixes, you need to understand what Mamba-2 traded away. Since the release of Mamba-2 in mid-2024, most architectures have switched from Mamba-1. Mamba-2 made the SSM recurrence simpler and more regular to maximize GPU throughput during training, achieving 2-8x faster training speeds. But that simplification had a hidden cost at inference time.
During token generation (decoding), each new token requires only a tiny state update. Mamba-3 is a bet that the age of inference has arrived. Rather than simplifying the SSM to train faster, it enriches the recurrence via trapezoidal discretization, complex-valued states, and MIMO to make each decode step do more useful work using GPU cycles that were previously idle. The GPU was spending most of its time moving memory, not computing, which is the worst possible situation for hardware utilization.
Three innovations, all from classical control theory
Mamba-3 proposes three major architectural advances: a new "exponential-trapezoidal" SSM discretization, complex-valued SSM state transitions, and a multi-input, multi-output (MIMO) SSM structure. Crucially, all three come from classical signal processing and control theory, not from the linear attention or test-time training frameworks that dominate most competing architectures.
1. Exponential-Trapezoidal Discretization
State space models are continuous-time systems that must be converted (discretized) to process discrete token sequences. Previous iterations like Mamba-1 and Mamba-2 utilized a first-order heuristic known as "exponential-Euler" discretization. Mamba-3 replaces this with exponential-trapezoidal discretization, which provides a second-order accurate approximation of the state-input integral. Think of it as the difference between a left-endpoint Riemann sum and the trapezoidal rule in calculus: the latter is simply more accurate for the same compute cost. A useful side effect: this new discretization implicitly applies a convolution inside the recurrence, which lets Mamba-3 drop the explicit short causal convolution that Mamba-1 and Mamba-2 both required.