Moonshot AI's Kimi-VL Matches 70B Reasoning While Activating Only 2.8B Parameters
Moonshot AI's Kimi-VL-A3B-Thinking packs 16B total parameters into a 2.8B-active MoE model that rivals 70B-scale VLMs on math and visual reasoning benchmarks
PRO- Kimi-VL-A3B-Thinking is a free, MIT-licensed open-source VLM with 16B total parameters but only 2.8B active at inference time via a MoE architecture.
- On MathVision, it scores 36.8 — matching or beating models like QVQ-72B (35.9) and GPT-4o (30.4) at a fraction of the compute cost.
- Its custom vision encoder, MoonViT, processes images at native resolution without sub-image splitting, enabling strong OCR (83.2 on InfoVQA) and high-res understanding.
- The model supports a 128K context window and excels at long video (74.2% on MLVU-MCQ) and long document tasks (35.1 on MMLongBench-Doc).
- An updated 2506 version is already available, with +20.1 on MathVision and 20% shorter thinking chains.
- Available via Hugging Face, with vLLM and SGLang serving support; use Temperature=0.8 for the Thinking variant.
Most open-source vision-language models force a painful tradeoff: small models are cheap to run but weak at reasoning, while capable models require massive GPU clusters. Moonshot AI's Kimi-VL-A3B-Thinking is a direct attack on that tradeoff. It uses a Mixture-of-Experts (MoE) architecture , where only a subset of the model's weights are active for any given input , to deliver 16B total parameters while activating only 2.8B at inference time. The result is a model that thinks hard without burning through compute.
Kimi-VL is an efficient open-source MoE vision-language model that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities, all while activating only 2.8B parameters in its language decoder. The Thinking variant specifically is the reasoning-focused version, trained with long chain-of-thought (CoT) supervised fine-tuning and reinforcement learning on top of the base model.
The architecture doing the heavy lifting
Three components make up Kimi-VL's architecture, and each one is doing something non-trivial:
- MoonViT , a native-resolution vision encoder. MoonViT processes images at varying resolutions without complex sub-image splitting. It uses a packing method from NaViT where images are divided into patches, flattened, and concatenated into 1D sequences, allowing MoonViT to share core computation operators with language models, including the variable-length sequence attention mechanism supported by FlashAttention.
- MLP Projector , a two-layer projector that converts MoonViT image features into the language-model embedding space.
- MoE Language Decoder , a Moonlight-based Mixture-of-Experts language model with 16B total parameters and about 2.8B activated parameters. Its architecture is similar to DeepSeek-V3.
MoonViT also integrates 2D rotary positional embedding (RoPE) across height and width dimensions to enhance fine-grained positional information, especially in high-resolution images. This matters because most vision encoders use fixed-size positional embeddings that degrade at higher resolutions , MoonViT avoids this entirely.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
