Tsinghua's Puro-2B Trains a Language Model for Less Than One GPU Rental
A Tsinghua team trained a 2B language model from scratch on consumer RTX 5090s for under $6,900, matching Qwen2.5-1.5B and open-sourcing everything.
PRO- Puro-2B matches Qwen2.5-1.5B trained for under $6,900 on RTX 5090s, versus $1.5M for Llama-3.2-3B.
- Recipe combines RTX 5090 hardware, blockwise FP8, MuonH optimizer, and Curriculum Model Averaging with 73% MFU.
- Team modified NVIDIA drivers to enable PCIe P2P and GPUDirect RDMA on consumer GPUs, doubling AllReduce bandwidth.
- Full release includes weights, ten checkpoints, training code, and data pipeline at the Puro-2B collection.
- A fitted cost scaling law says $4.4K is enough to cross Qwen2-1.5B under the same recipe.
- Curriculum pretraining advantage survives SFT: +1.77 pp GSM8K focused, +1.59 pp broad instruction macro-average.
Pretraining a language model from scratch has become a rich lab's game. A new report from Tsinghua's PACMAN group pushes back with Puro-2B, a 2-billion-parameter dense Transformer trained on 1.4 trillion tokens for less than the price of renting a single RTX 5090 GPU. The team's best checkpoint approaches Qwen2.5-1.5B on their evaluation, and their scaling law suggests you can match Qwen2-1.5B for roughly $4,400.
The claim rests on a specific accounting boundary worth pinning down. Reproduction cost here refers only to the compute cost of rerunning the finalized two-phase pretraining recipe once, converted from measured GPU-hours at a normalized RTX 5090 rental-equivalent rate. Data acquisition, proxy experiments, failed runs, and research labor are excluded. Even with that caveat, the delta versus prior open-recipe efforts is dramatic.
The cost gap this closes
Open-weight releases like Qwen, Gemma, and Llama publish checkpoints but withhold the data ordering, exact recipe, and training state needed to reproduce a run. Open-recipe projects, including OLMo, SmolLM, Yulan-Mini, and Instella, expose the full pipeline, yet the compute they assume is still out of reach for most academic labs. Under the report's rental-equivalent accounting, training Llama3.2-3B costs over $1.5M, reproducing OLMoE-1B-7B would cost $200K, and reproducing SmolLM3-3B rises to $719K.
Puro-2B tries to close the last mile between reproducibility and accessibility. The best checkpoint uses 438.8B Phase 1 tokens and 960.0B Phase 2 tokens, totaling 1.4T scheduled tokens and 22,514 active-training GPU-hours, corresponding to a compute cost of about $6.9K and 17.6 elapsed days.
Why consumer GPUs actually win here
The most contentious choice is the hardware. RTX 5090s carry 32 GB of memory, no NVLink, and NVIDIA has software-disabled several data-center features. The price, however, is different by an order of magnitude. On the authors' pricing, an 8-GPU node costs roughly $1,763 per month, amortizing to about $0.31 per GPU-hour compared with $4.00 for an H200.
That translates to a clear compute-per-dollar advantage, especially with FP8:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
