AI2 Opens OLMo-core, the Exact Engine Behind Its Frontier Models
Allen AI's OLMo-core is a fully open PyTorch library for training frontier LLMs, now powering OLMo-2 and OLMo-3 at 40%+ GPU efficiency
PRO- OLMo-core is AI2's open-source PyTorch library for training large language models, used to build OLMo-2 and OLMo-3, available free under Apache 2.0.
- Achieves 43% MFU on 7B models and 41% on 32B models on H100s, with 7,700 tokens/sec/GPU at 8K sequence length.
- Supports FSDP, tensor parallelism, context parallelism, and expert parallelism for MoE models, configurable in a few lines of Python.
- Moving SFT to OLMo-core gave AI2 an 8x throughput increase; RL training became 4x more efficient for OLMo-3.
- Covers pretraining, midtraining, long-context extension, and SFT; DPO and RL support are planned but not yet available.
- Install via
pip install ai2-olmo-core; official training scripts for OLMo-2 and OLMo-3 are included in the GitHub repo.
OLMo-core is the Allen Institute for AI's (AI2) open-source training engine for large language models, and it's quietly become one of the most complete, production-grade LLM training stacks available to the public. While most labs keep their training infrastructure proprietary, AI2 ships theirs as a pip-installable Python library, Apache 2.0 licensed, with official training scripts for their frontier models included.
The problem it solves
Building a serious LLM training setup from scratch is brutal. You need to wire together distributed parallelism strategies, memory-efficient attention kernels, checkpoint management, data pipelines, and evaluation loops, all while keeping things reproducible across hundreds of GPUs. Most teams either bolt together fragile scripts or pay for a managed platform. OLMo-core represents a major rewrite of the original training and modeling code from OLMo with a focus on performance and API stability. The goal: give researchers a standard, battle-tested toolkit they can actually build on.
What's inside
OLMo-core is a Python library that provides building blocks for large-scale distributed training with PyTorch. The library is organized around a few key abstractions that handle the heavy lifting:
- TransformerTrainModule -- a single object that wraps both the model and optimizer, managing all parallelism config in one place
- Trainer -- the main training loop, with async metric logging and a flexible callback system for extending behavior
- Data pipeline -- composable dataset mixtures with reproducible ordering, remote directory support, and built-in data mixes like Dolma
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
