LSTM Co-Inventor's NXAI Releases KronLSTM to Beat Long-Context Memory Limits
A new xLSTM variant corrects one scalar instead of a whole vector per memory update, cutting interference and improving long context performance.
- NXAI released KronLSTM, a linear attention model built on a new Kronecker Delta update rule.
- Updates revise a single scalar key-value binding instead of a full vector, cutting memory interference.
- Memory dynamics are provably bounded, removing the need for extra normalizers and stabilizers.
- Beats Gated DeltaNet on NIAH at 32k context in both pure and SWA-hybrid configurations.
- Four 1.5B checkpoints released on Hugging Face, pretrained on 100B tokens.
- Code is open source at github.com/NX-AI/KronLSTM, co-authored by LSTM creator Sepp Hochreiter.
NXAI researchers, including LSTM co-inventor Sepp Hochreiter, have introduced the KronLSTM paper, which proposes a narrower way to revise matrix memory in recurrent linear-attention models. KronLSTM adjusts the scalar coupling between a key and a value, limiting interference with other stored associations. The reported gains are strongest on long-context retrieval and state-tracking benchmarks.
- Core change: The Kronecker Delta Rule constrains each memory correction to one direction in matrix space.
- Practical goal: Preserve earlier associations when overlapping keys receive new values.
- Reported evidence: Higher 32K retrieval scores than Gated DeltaNet-2, plus competitive state-tracking results and comparable training throughput.
- Current limit: Published checkpoints reach 1.5 billion parameters and 100 billion training tokens.
Why matrix memories collide
Standard self-attention forms pairwise interactions between tokens, giving it quadratic cost as sequence length grows. Linear attention compresses prior tokens into a fixed-size recurrent state, allowing sequence processing to scale linearly with length. That efficiency makes the quality of each memory update especially important.
Hebbian and delta updates expose the trade-off between accumulation and revision. A Hebbian update adds a key-value contribution to memory at every token, so old values can be buried by later writes but cannot be directly corrected. A delta update reads the value associated with a key, subtracts that full d-dimensional vector, and writes a replacement, where
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
