Stanford's Marin Trains a 535B Open Model Anyone Can Watch Live

Open foundation model training just got its most transparent large-scale run yet. The Marin team, an open lab out of Stanford CRFM, has started training a 535B-parameter mixture-of-experts model with 23B active parameters, and unlike almost every other frontier-scale run in the industry, you can watch the loss curves tick by in real time on Weights & Biases.

Marin is an open-source framework for training foundation models where every step from raw data to final weights is recorded, including failed experiments. The lab previously used it to train an 8B model that beat Llama 3.1 8B and a 32B dense model. The new 535B-A23B run, nicknamed the Hero Run, is by far its largest attempt.

The voyage plan

Percy Liang described the launch on X, laying out an unusually detailed spec for a training run in progress. Here is what the team committed to publicly:

• Architecture: 535B total parameters, 23B active per token (a mixture-of-experts design where only a small subset of experts fire on each forward pass)
• Data budget: 18.75T tokens split 80% pretraining and 20% midtraining
• Hardware: 11 x GB200 NVL72 racks for roughly 3 months
• Compute: 2.7e24 FLOPs total
• Post-training: to follow after the pretraining voyage completes

Before committing 100 days of GB200 time, the team ran what they call a 4-rung scaling ladder from 1.6B-A61M on 48B tokens up to 27.7B-A1.2B on 926B tokens to debug issues and forecast the hero run. The ladder costs about 1% of total compute but yields a projection curve to check against throughout the run.

A ladder that catches falls

The ladder doubles as an active debugging tool. Across the entire ~100 day run, the team knows exactly how they are doing compared to the projection, and any deviation prompts early investigation into training dynamics. That approach has already paid off in prior runs.