AI2 Opens OLMo 3's Full 32B Training Stack Where Qwen3 Stayed Closed

OLMo-core is the PyTorch training library that the Allen Institute for AI (AI2) uses to build and release its OLMo model family. Unlike most training frameworks that stay internal, AI2 ships OLMo-core as a fully open-source library under Apache 2.0, complete with the exact training scripts, data mixes, and configs used to produce their released models. It is, in effect, the engine behind every OLMo model you can download today.

More than a training loop

OLMo-core represents a major rewrite of the original training and modeling code from OLMo, with a focus on performance and API stability. It aims to provide a standard set of robust tools that LLM researchers at AI2 and other organizations can use to build their research projects on, centered around a highly efficient yet flexible Trainer and a launch module that handles all the boilerplate of launching experiments on Beaker or other platforms.

The library is structured around a clean config-then-build pattern. You define your experiment through typed config classes, then call .build() on each component at runtime. This makes experiments reproducible, serializable, and easy to override from the command line.

model_config = TransformerConfig.llama2_7B(...)
train_module_config = TransformerTrainModuleConfig(...)
data_config = NumpyFSLDatasetConfig(...)
trainer_config = TrainerConfig(...)
def main():
    model = model_config.build()
    train_module = train_module_config.build(model)
    trainer = trainer_config.build(train_module, data_loader)
    trainer.fit()

The parallelism stack is production-grade

Scaling LLMs across hundreds of GPUs requires combining multiple parallelism strategies simultaneously. OLMo-core ships all of them in one coherent API. The TransformerTrainModule supports three dimensions of parallelism for dense models: data parallelism through FSDP or DDP, tensor parallelism (TP), and context parallelism (CP), as well as expert parallelism (EP) for MoE models.