Ai2's OLMo 2 1B Beats Llama and Gemma With Full Open Training Recipe
Ai2 rounds out the OLMo 2 family with a fully open 1B model trained on 4T tokens that beats Gemma 3 1B and Llama 3.2 1B.
PRO- Ai2 released OLMo 2 1B, the smallest fully open member of the OLMo 2 family, under Apache 2.0.
- Pretrained on 4T tokens with 16 layers, 2048 hidden size, and a 4096 context window.
- Averages 42.7 across benchmarks, beating Gemma 3 1B, Llama 3.1 1B, and Qwen 2.5 1.5B.
- Strong at instruction following (IFEval 70.1) and GSM8K (68.3), weaker at MMLU and MATH.
- Ships base, SFT, DPO, and GRPO instruct variants plus hundreds of intermediate checkpoints.
- Full recipe documented in the 2 OLMo 2 Furious paper.
OLMo 2 1B is Ai2's latest addition to the OLMo 2 family: a fully open, Apache 2.0 licensed base model that outscores Gemma 3 1B, Llama 3.2 1B, and Qwen 2.5 1.5B on aggregate benchmarks, while shipping the complete training recipe, data, and intermediate checkpoints alongside the weights.
What's in the box
The model is a decoder-only transformer with 16 layers, a hidden size of 2048, 16 attention heads, and a 4096-token context window. Pretrained on 4 trillion tokens, it's heavily overtrained for its size, which is much of the reason it holds its own against larger competitors. Ai2 released a full post-training lineup alongside the base model:
- Base: OLMo-2-0425-1B
- SFT: supervised fine-tuned checkpoint on instruction data
- DPO: preference-tuned via Direct Preference Optimization
- Instruct (final): further refined with GRPO, the group relative policy optimization method popularized by DeepSeek
All variants and hundreds of intermediate pretraining checkpoints are on Hugging Face. Naming follows the convention stage1-stepXXX-tokensYYYB, so you can load the model at, say, 294B tokens of pretraining and study how capabilities emerge over time.
Benchmark numbers, including the weak spots
Ai2 reports the instruct model averaging 42.7 across a ten-benchmark suite, compared with 41.7 for Qwen 2.5 1.5B, 39.3 for Llama 3.1 1B, and 38.3 for Gemma 3 1B. The gap is widest on math and instruction following: GSM8K hits 68.3 and IFEval hits 70.1. Safety scores at 87.6, on par with much larger open models.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
