Complex KDA Unlocks Rotational Memory in Linear AI Without Extra Cost
A two-parameter tweak to Kimi Delta Attention unlocks 2D rotations in one layer, matching DeltaProduct's expressivity without extra cost.
Takeaways
- Complex KDA extends Kimi Delta Attention gate range to [-1,1] and β to [0,2] to unlock 2D rotations.
- A single CKDA layer can track any finite group inside SO(3), including S₃, S₄, and A₅.
- Matches DeltaProduct₂ expressivity while keeping cheaper single diagonal-plus-rank-one transitions.
- Strongest length extrapolation among KDA variants on S₃, S₄, and periodic audio continuation tasks.
- Beats Transformers and other linear RNNs in language modeling with promising scaling behavior.
- Open source at GitHub with models on Hugging Face.
Kimi Linear and Qwen3-Next use delta-rule variants as primary token mixers, reducing sequence-mixing work to linear in sequence length and decoding with a fixed-size recurrent state. Those efficient updates also impose a single-layer expressivity limit: existing parameterizations cannot represent some simple operations, including rotation of a two-dimensional vector. The Complex KDA paper expands two coefficient ranges in Kimi Delta Attention, enabling rotations without adding recurrent state, increasing matrix rank, or introducing complex-valued arithmetic.
Why the original spectrum misses rotations
Softmax attention decoders retain a key-value cache that grows with the context. Linear-attention models compress prior tokens into a fixed-size recurrent state, trading a larger cache for a structured state update at each token.
DeltaNet updates that state with the delta rule, which uses prediction error to overwrite the association addressed by the current key. Gated DeltaNet adds a learned scalar decay that controls global forgetting. Kimi Delta Attention, or KDA, assigns a separate gate to each channel, allowing individual feature dimensions to decay at different rates.
Each token applies a diagonal-plus-rank-one transition, meaning a diagonal channel update receives one outer-product correction. This structure is inexpensive to evaluate. Under the original gate and delta-coefficient ranges, however, the relevant eigenvalues remain real. A single layer therefore cannot produce the complex-conjugate eigenvalues associated with rotation in a two-dimensional subspace.