Skip to main content
Back

Complex KDA Unlocks Rotational Memory in Linear AI Without Extra Cost

A two-parameter tweak to Kimi Delta Attention unlocks 2D rotations in one layer, matching DeltaProduct's expressivity without extra cost.

  • Complex KDA extends Kimi Delta Attention gate range to [-1,1] and β to [0,2] to unlock 2D rotations.
  • A single CKDA layer can track any finite group inside SO(3), including S₃, S₄, and A₅.
  • Matches DeltaProduct₂ expressivity while keeping cheaper single diagonal-plus-rank-one transitions.
  • Strongest length extrapolation among KDA variants on S₃, S₄, and periodic audio continuation tasks.
  • Beats Transformers and other linear RNNs in language modeling with promising scaling behavior.
  • Open source at GitHub with models on Hugging Face.

Kimi Linear and Qwen3-Next use delta-rule variants as primary token mixers, reducing sequence-mixing work to linear in sequence length and decoding with a fixed-size recurrent state. Those efficient updates also impose a single-layer expressivity limit: existing parameterizations cannot represent some simple operations, including rotation of a two-dimensional vector. The Complex KDA paper expands two coefficient ranges in Kimi Delta Attention, enabling rotations without adding recurrent state, increasing matrix rank, or introducing complex-valued arithmetic.

Why the original spectrum misses rotations

Softmax attention decoders retain a key-value cache that grows with the context. Linear-attention models compress prior tokens into a fixed-size recurrent state, trading a larger cache for a structured state update at each token.

DeltaNet updates that state with the delta rule, which uses prediction error to overwrite the association addressed by the current key. Gated DeltaNet adds a learned scalar decay that controls global forgetting. Kimi Delta Attention, or KDA, assigns a separate gate to each channel, allowing individual feature dimensions to decay at different rates.

Each token applies a diagonal-plus-rank-one transition, meaning a diagonal channel update receives one outer-product correction. This structure is inexpensive to evaluate. Under the original gate and delta-coefficient ranges, however, the relevant eigenvalues remain real. A single layer therefore cannot produce the complex-conjugate eigenvalues associated with rotation in a two-dimensional subspace.

Two wider ranges create the missing spectrum