MegaTrain Trains 100B Models on a Single GPU 5.93x Faster
MegaTrain trains 120B-parameter models on a single H200 GPU by inverting the memory hierarchy, outperforming DeepSpeed ZeRO-3 by 1.84x on 14B models
ProTakeaways
- MegaTrain trains 100B+ parameter LLMs at full precision on a single GPU by storing all weights in CPU RAM and streaming them to the GPU one layer at a time.
- Achieves 1.84x the throughput of DeepSpeed ZeRO-3 on 14B models and sustains 227-284 TFLOPS from 28 to 180 layers on a GH200.
- Trains a 120B parameter model on a single H200 with 1.5TB host memory; competitors hit OOM beyond 32B.
- Two key innovations: a pipelined double-buffered execution engine across 3 CUDA streams, and stateless layer templates that replace PyTorch's global autograd graph.
- Works on consumer hardware -- trains 14B models on an RTX 3090 (24GB VRAM) where ZeRO-3 runs out of memory entirely.
- Fully open-source at github.com/DLYuanGod/MegaTrain with SFT and GRPO/RLHF support included.
Training a 100-billion-parameter model has always meant one thing: a cluster of GPUs and a budget to match. MegaTrain, a new open-source training system from researchers at the University of Notre Dame and Lehigh University, challenges that assumption head-on. By flipping the conventional memory hierarchy upside down, it makes full-precision training of 100B+ parameter models possible on a single GPU -- and does it faster than the leading alternatives.
The Wall Every Practitioner Hits
A 100B parameter model in full FP32 precision needs roughly 400GB just for the parameters. Add optimizer states (Adam stores two additional copies), gradients, and activations -- you are looking at well over a terabyte of memory. A single H200 has 141GB of HBM. The math simply does not work.
Existing offloading techniques like ZeRO-Offload and ZeRO-Infinity have begun to extend GPU capacity by migrating model states to host memory and NVMe storage, but they do not fully exploit the memory hierarchy: the GPU remains the host of model parameters, while CPU and storage serve only as temporary spill buffers. In training, parameters, gradients, and optimizer states are accessed infrequently relative to activations, yet they remain persistently pinned in device memory.
As model sizes grow, this underutilization of the memory hierarchy makes it increasingly difficult for end users, small companies, and research labs to train large models on their own hardware. The paper even cites a striking data point: among 167 U.S. universities, only two achieve an average availability of more than one H100 per student.
Inverting the Hierarchy
MegaTrain's core insight is a complete role reversal. Unlike traditional GPU-centric systems, MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, it streams parameters in and computes gradients out, minimizing persistent device state. The GPU never holds more than one layer's worth of weights at a time.

This design means the memory-centric training math relies on a baseline of roughly 12 bytes per parameter stored in host DDR5 RAM -- which costs roughly $5-12/GB versus ~$20/GB for HBM. Host memory becomes the authoritative store; the GPU is just a very fast calculator that borrows weights one layer at a time.
The catch is obvious: constantly shuttling weights between CPU RAM and GPU over PCIe is slow. MegaTrain addresses this with two interlocking innovations.