Moonshot AI's Kimi K3 Becomes the World's First Open 2.8T Model

Moonshot AI has released the weights for Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model the company calls the world's first open 3T-class model. It ships with native vision, a 1-million-token context window, and a new attention architecture that Moonshot claims delivers 2.5x better scaling efficiency than its predecessor, Kimi K2.

What "2.8 trillion parameters" actually means

K3 is a sparse Mixture-of-Experts (MoE) architecture, so only a fraction of those 2.8 trillion weights activate for any given token. Each token routes through 16 of the model's 896 expert modules, putting the active parameter count at roughly 50 billion per forward pass. That distinction matters for hardware planning and for comparing K3 against dense models of similar nominal size.

Three architectural bets under the hood

The parameter count is the headline, but the more interesting story is the three architectural changes Moonshot built to make this scale work efficiently.

• Kimi Delta Attention (KDA): A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers. KDA preserves full expressiveness where it counts while cutting the computational cost of attention across the 1M-token context window, delivering 6.3x faster decoding at that context length.
• Attention Residuals (AttnRes): Rather than uniformly accumulating state through every layer, AttnRes selectively retrieves useful representations from earlier in the network. The result is smoother information flow through a very deep model, with Moonshot reporting 25% training efficiency gains at 2% additional compute cost.
• Stable LatentMoE: The framework that routes tokens to 16 of 896 experts. Its key sub-innovation, Quantile Balancing, derives expert allocation directly from router-score quantiles, removing the fragile hyperparameters that typically destabilize MoE training at scale.

Together, these changes give K3 roughly 2.5x the scaling efficiency of K2, converting compute into capability more effectively. K3 also applies quantization-aware training from the supervised fine-tuning stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility. The native MXFP4 weight release weighs in at around 594GB.

Where K3 leads

K3 does not top every benchmark, but it leads on the tasks most relevant to agentic and long-horizon work.

• Ranked first on the Frontend Code Arena at Arena.ai with 1,679 Elo, ahead of Claude Fable 5 (1,631), GPT-5.6 Sol (1,618), and GLM-5.2 (1,587).
• On SWE Marathon, which tests sustained multi-step reasoning over extended codebases, K3 scores 42.0, beating Opus 4.8 (40.0), GPT-5.6 (39.0), and Fable 5 (35.0).
• Leads on BrowseComp (91.2), DeepSearchQA (95.0 F1), and AutomationBench (30.8), three agentic evaluations covering information navigation, data synthesis, and multi-step workflow execution.
• Tops Program Bench (77.8) above all compared models, including Fable 5 and GPT-5.6 Sol.
• Scores 93.40% on SWE-bench Verified.
• Scores 57 on the Artificial Analysis Intelligence Index, placing fourth overall and first among open-weight models.

Where it falls short

Moonshot is candid about the gaps. K3 scores 67.5 on the DeepSWE coding benchmark, trailing GPT-5.6 Sol (73.0) and Claude Fable 5 (70.0). Fable 5 leads by 10 points on HLE-Full reasoning. Conversational polish also lags behind the top proprietary models in subjective use.

There are also behavioral quirks worth knowing before deployment:

• Excessive proactiveness: In ambiguous scenarios, K3 tends to act rather than ask for clarification. Applications that require the agent to stay within defined boundaries need explicit system prompt instructions.
• Thinking history dependency: K3 was trained in preserved thinking history mode. If the agent harness does not pass back all historical thinking content, generation quality degrades significantly. Use a compatible harness like Kimi Code.

Long-horizon demos worth examining

Moonshot published several autonomous case studies that go beyond benchmark tables. In a 24-hour kernel optimization session, K3 rewrote and benchmarked GPU kernels across NVIDIA Hopper and alternative-vendor hardware, then built MiniTriton, a compact compiler with its own tile-level intermediate representation over MLIR, optimization passes, and PTX generation. On supported roofline workloads, MiniTriton matched or beat Triton, and nanoGPT trained stably through the resulting stack.

In a chip design experiment running 48 hours autonomously, K3 produced a 4mm² design with 1.46M standard cells, 0.277MB of SRAM, and an INT4 MAC array, closing timing at 100MHz. In a computational astrophysics task, K3 implemented a full numerical pipeline across 300+ equations of state in roughly two hours, a task Moonshot estimates would take an experienced researcher one to two weeks.

Infrastructure released alongside the weights

Moonshot is open-sourcing parts of the stack that made K3 possible: high-performance attention kernels, a MoE communication library, and tooling for running agent environments at scale. Teams building large MoE systems will find these worth examining.

The vLLM team published a preview of their day-0 serving support, which required substantial new engineering. The core challenge was prefix caching for KDA. Because KDA reduces conventional KV pairs but introduces a large recurrent state, it breaks the assumptions underlying standard prefix caching. vLLM redesigned its cache manager to separate physical block size from prefix-match granularity, enabling partial cache hits without storing the full recurrent state at every boundary. Moonshot has contributed a KDA prefill-cache implementation to the vLLM project. Day-0 support is a joint effort involving NVIDIA (KDA decode and AttnRes kernels), AMD (FlyDSL MLIR kernel stack), and Inferact.

How to run it today

• API: Call kimi-k3 on the Kimi API Platform. Pricing is $0.30/MTok for cache-hit input, $3.00/MTok for cache-miss input, and $15.00/MTok for output. These rates are 70–80% cheaper than Claude Opus 4.8 on output tokens. Moonshot reports a cache hit rate above 90% in coding workloads.
• Web and mobile: Available on kimi.com, iOS, Android, and HarmonyOS. K3 uses maximum thinking effort by default.
• Kimi Code: Run it in your terminal and select K3 with the /model command.
• Open weights: Expect a ~594GB download for the native MXFP4 safetensors release. Community BF16 and GGUF re-quants will likely follow within days.

Self-hosting is a serious infrastructure commitment. Moonshot recommends supernode configurations with 64 or more accelerators. KDA is not yet supported in llama.cpp, Ollama, or LM Studio, and Stable LatentMoE routing requires custom kernel support. For most teams, the API is the practical starting point.

What this means for the open ecosystem

K3's release drew immediate comparisons to earlier DeepSeek moments from engineers and model-watchers, and the comparison has merit. Open weights get distilled, quantized, and optimized fast, as happened with Llama, Qwen, and DeepSeek, and each cycle moves faster than the last.

KDA, AttnRes, and Stable LatentMoE are now available for the community to study and build on. If the 2.5x scaling efficiency claim holds under independent scrutiny, it suggests that reaching frontier capability does not require simply scaling parameter counts. That is a useful signal for anyone designing the next generation of large models.