Moonshot AI's Kimi K3 Becomes the World's First Open 2.8T Model

Moonshot AI just dropped the weights for Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model that the company is calling the world's first open 3T-class model. It ships with native vision, a 1-million-token context window, and a new attention architecture that Moonshot says delivers 2.5x better scaling efficiency than its predecessor, Kimi K2. This is not a minor version bump.

The headline number is a bit misleading

When Moonshot says 2.8 trillion parameters, they mean the total number of weights stored in the model. But K3 is a sparse Mixture-of-Experts (MoE) architecture, meaning only a small fraction of those weights are actually used for any given token. K3 has 2.8 trillion total parameters, but only 16 of its 896 experts are active per token , meaning roughly 50 billion parameters are being used in each forward pass. That distinction matters enormously for hardware requirements and for understanding what "2.8T" actually means in practice.

Each token is routed through only 16 of the 896 expert modules. The active parameter count per token is a small fraction of 2.8T. This is typical MoE design, but it means "2.8 trillion parameter model" is not comparable to a hypothetical 2.8T dense model.

Two new architectural ideas under the hood

The real story is not the parameter count , it's the two architectural innovations Moonshot built to make this scale work efficiently.

• Kimi Delta Attention (KDA): A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers. KDA maintains full expressiveness for critical layers while reducing the computational cost of attention across the model's 1M-token context window. The payoff is significant: KDA delivers 6.3x faster decoding at 1M context.
• Attention Residuals (AttnRes): Moonshot describes AttnRes as selectively retrieving representations across depth rather than uniformly accumulating everything. The intended benefit is smoother information flow through a very deep model , instead of forcing every layer to depend only on the immediately accumulated state, AttnRes can emphasize useful earlier representations. Attention Residuals add 25% training efficiency at 2% compute cost.
• Stable LatentMoE: With the Stable LatentMoE framework, the model efficiently activates 16 out of 896 experts. A key sub-innovation here is Quantile Balancing, which derives expert allocation directly from router-score quantiles, eliminating the fragile hyperparameters that typically break MoE models at scale.