Yandex Drops AliceAI Foundation 80B Open Source Model That Beats Rivals on Math
Yandex open-sourced an 80B mixture-of-experts model with 3B active parameters, 256K context, and a hybrid KDA-attention architecture trained from scratch.
- Yandex released AliceAI-Foundation-80B-A3B-Base under Apache 2.0, ungated on Hugging Face
- Hybrid MoE: 80B total, 3B active per token, 256K context window
- Novel layer schedule interleaves KDA linear attention with Gated Attention plus 512-expert MoE
- Trained entirely from scratch, no Qwen or Llama initialization
- Tops MATH-500 (91.1), HMMT, IMO Answerbench, and Russian factual benchmarks vs larger MoE rivals
- Base model only, not chat-tuned; ships LoRA fine-tuning example and vLLM Docker path
Yandex releases AliceAI Foundation 80B-A3B under Apache 2.0
Yandex has released the weights for AliceAI Foundation, a bilingual Russian-English mixture-of-experts model trained from scratch. The checkpoint uses 80 billion parameters while activating about 3 billion for each token, supports a 262,144-token context window, and is available without an access gate under the Apache 2.0 license.
The release gives developers a commercially usable base model with an uncommon hybrid-attention architecture and relatively low active compute. Its sparse execution reduces arithmetic per token, while the full BF16 checkpoint still requires about 163 GB of storage before runtime overhead.
80B weights, 3B active
- Parameters: 80 billion total, approximately 3 billion active per token
- Context window: 262,144 tokens
- Network: 48 layers with a hidden size of 2,048
- Vocabulary: 129,024 tokens
- Experts: 512 routed experts, with 10 selected per token and 1 shared expert
- Checkpoint: Approximately 163 GB in BF16 across 49 safetensors files
- Languages: Russian and English
- License: Apache 2.0
- Decoding: Multi-token prediction support for speculative decoding
The reported active-to-total parameter ratio is about 3.75%. That sparsity lowers per-token computation, but deployment still requires enough memory for the complete weights, key-value caches, activations, routing, and framework overhead.
Two attention systems share the stack
AliceAI repeats a four-layer pattern 12 times instead of using one transformer block throughout the network. Each cycle contains three KDA blocks followed by one Gated Attention block, with an MoE feed-forward module attached to every layer.
KDA is a kernelized, linear-attention-style operator configured with a convolution kernel size of 4, 32 query heads, and 32 key-value heads. It is designed to process long sequences more cheaply than full attention. The Gated Attention layers use 16 query heads, 2 key-value heads, and a head dimension of 256, periodically adding a different attention mechanism to the stack.
Each MoE feed-forward contains 512 small experts with an intermediate dimension of 512. The router selects 10 experts for each token and also invokes one shared expert, distributing computation across many specialized parameter groups without activating the entire model.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
