Mamba-3’s Rotations Behave Like Damped Oscillators

Mamba-3 added input-dependent rotations to recurrent state updates, expanding the sequence computations that diagonal state space models can represent. Earlier Mamba-2-style transitions mainly scaled stored state down at each step. Their real, positive transition values support fading memory but exclude periodic state changes required for tasks such as parity and modular counting.

A new analysis, Understanding Mamba-3 Through Oscillatory Dynamics, examines how trained models use this capability. The paper finds that Mamba-3 combines rotation with decay to create underdamped oscillations. Controlled tasks produce phase structures aligned with their underlying arithmetic, while language-model experiments connect oscillation to stronger extrapolation beyond the training context.

Rotation gives memory a phase

In simplified terms, a Mamba-2-style transition multiplies the recurrent state by a scalar between zero and one. Each token therefore weakens previous state before adding new information. Mamba-3 introduces a complex-valued transition whose angle can depend on the current input.

The implementation avoids explicitly storing complex states. It expresses the transition through data-dependent rotary positional embedding, or RoPE, operations applied to the model’s B and C projections. The B projection controls how input enters the recurrent state, while C controls how that state contributes to the output.

A complex transition can be written as