Moonshot AI Open-Sources FlashKDA, Making Kimi Linear 2x Faster on GPUs

Moonshot AI just open-sourced FlashKDA, a high-performance CUDA kernel for Kimi Delta Attention (KDA) -- the linear attention mechanism at the heart of their Kimi Linear architecture. The repo is available now under an MIT license, free to use and extend.

This matters because KDA was already a strong architecture on paper, but its Triton-based implementation in flash-linear-attention was leaving significant GPU performance on the table. FlashKDA is the production-grade answer: a fused CUTLASS kernel that slots in as a drop-in backend with no model changes required.

Why KDA in the first place?

Kimi Linear is a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. That's a significant claim, and it's built on a specific architectural choice.

At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. The bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation.

To unpack that: standard softmax attention is expensive because every token attends to every other token -- cost scales quadratically with sequence length. Linear attention replaces this with a recurrent state that's updated token-by-token, making cost linear. The tradeoff is that a fixed-size state can't perfectly remember everything. KDA's fine-grained gating is designed to make that fixed memory budget go further by controlling, channel by channel, what gets written in and what gets forgotten.