Moonshot AI's Kimi K3 Becomes the World's First Open 2.8T Model
Moonshot AI releases Kimi K3, a 2.8T open-weight MoE model with a 1M-token context window that beats frontier models on sustained coding and agentic tasks
- Kimi K3 weights are now live on Hugging Face — the largest open-weight model ever released at 2.8T parameters.
- New architecture: Kimi Delta Attention (KDA) enables 6.3x faster decoding at 1M-token context; Attention Residuals add 25% training efficiency at 2% extra compute cost.
- Benchmark wins: #1 on Frontend Code Arena, SWE Marathon, BrowseComp, and Program Bench — beating Claude Fable 5 and GPT-5.6 Sol on sustained agentic tasks.
- Gaps remain: Trails GPT-5.6 Sol on DeepSWE (67.5 vs 73.0) and Fable 5 on HLE reasoning; conversational UX still lags proprietary models.
- API pricing: $0.30/MTok cache-hit input, $3.00/MTok cache-miss input, $15.00/MTok output — 70-80% cheaper than Claude Opus 4.8 on output tokens.
- Self-hosting requires serious infrastructure: Moonshot recommends 64+ accelerators; KDA requires custom kernels not yet in llama.cpp or Ollama.
Moonshot AI has released the weights for Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model the company calls the world's first open 3T-class model. It ships with native vision, a 1-million-token context window, and a new attention architecture that Moonshot claims delivers 2.5x better scaling efficiency than its predecessor, Kimi K2.
What "2.8 trillion parameters" actually means
K3 is a sparse Mixture-of-Experts (MoE) architecture, so only a fraction of those 2.8 trillion weights activate for any given token. Each token routes through 16 of the model's 896 expert modules, putting the active parameter count at roughly 50 billion per forward pass. That distinction matters for hardware planning and for comparing K3 against dense models of similar nominal size.
Three architectural bets under the hood
The parameter count is the headline, but the more interesting story is the three architectural changes Moonshot built to make this scale work efficiently.
- Kimi Delta Attention (KDA): A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers. KDA preserves full expressiveness where it counts while cutting the computational cost of attention across the 1M-token context window, delivering 6.3x faster decoding at that context length.
- Attention Residuals (AttnRes): Rather than uniformly accumulating state through every layer, AttnRes selectively retrieves useful representations from earlier in the network. The result is smoother information flow through a very deep model, with Moonshot reporting 25% training efficiency gains at 2% additional compute cost.
- Stable LatentMoE: The framework that routes tokens to 16 of 896 experts. Its key sub-innovation, Quantile Balancing, derives expert allocation directly from router-score quantiles, removing the fragile hyperparameters that typically destabilize MoE training at scale.