Marin Opens 61 TB of Mochi 535B Training Snapshots to Researchers
Open Athena dropped all 57 pretraining checkpoints from its 535B mixture-of-experts run, totaling 61 TB and 18.2T tokens of training history.
Takeaways
- Open Athena released 57 intermediate pretraining checkpoints for Marin Mochi 535B-A23B, totaling 61 TB.
- Model is a 535B-parameter MoE with ~23B active, 384 experts (8 selected), trained on 18.2T tokens.
- Context extended across 4K, 16K, 65K, and 262K phases, all captured in the checkpoint branches.
- Architecture uses latent MoE with RMSNorm, sliding-window GQA, short convolutions, and exclusive self-attention.
- Team disclosed benchmark contamination that their decontamination procedure missed, limiting fair comparisons.
- Base model under OpenMDW 1.1, requires the Marin vLLM fork to serve.
Marin has released every retained checkpoint from the training of Marin Mochi 535B-A23B, giving researchers 57 snapshots of how a frontier-scale mixture-of-experts model changed across 18.2 trillion training tokens. The collection exposes capability emergence, representation shifts, context extension, and training stability at a scale rarely available outside major AI labs.
The model repository contains 535 billion total parameters, while roughly 23 billion participate in each token’s forward pass. That sparse design reduces inference work compared with activating the full model.
57 snapshots across 18.2T tokens
The release includes 57 intermediate checkpoints, 10 of them from long-context extension, totaling about 61 TB of weights. Each snapshot occupies a Git branch named step-N, with retained steps ranging from step-6000 to step-393491.
The checkpoint formats change as the context window grows:
| Training phase | Context window | Checkpoint detail |
|---|---|---|
| Early pretraining | 4K tokens | Initial retained snapshots |
| First extension | 16K tokens | Longer-context training |
| Second extension | 65K tokens | Several retained checkpoints |
| Final extension | 262K tokens | Steps 390500 through 393491 |
The default main branch contains an averaged model assembled from 11 late-stage snapshots. Marin used hyperball norm rescaling, its norm-aware method for merging the weights.
What Mochi computes per token
Mochi combines sparse expert routing, local and global attention, learned convolutions, and several uncommon positional choices. Marin’s architecture notes describe the main components:
- Core dimensions: 48 layers, a hidden width of 6,144, and 48 query-attention heads with a head dimension of 128.
- Expert routing: 384 routed experts, eight selected for each token, plus two shared experts. Routing passes through a 3,072-wide latent space with an additional RMSNorm that improved the latent MoE in Marin’s experiments.