Chinese Academy of Sciences' Janus Trains 11.5B Atom-Simulating AI at 1.2 EFLOPS
A new framework trains a 11.5-billion parameter atomistic foundation model on two exascale supercomputers, hitting 1.2 EFLOPS and cutting weeks of training down to hours.
Takeaways
- MatRIS-MoE scales invariant uMLIPs to 11.5B parameters using element-based expert routing.
- Janus framework unifies FSDP, graph parallelism, and expert parallelism for second-order training.
- Achieves 1.2 EFLOPS peak at 90%+ parallel efficiency across two exascale supercomputers.
- Training time for billion-parameter uMLIPs compressed from weeks to hours.
- Matches or beats UMA, MACE-mh, and SevenNet-Omni across molecules, materials, MOFs, and catalysis.
- Just-in-time expert planning avoids all-gathering unused experts, cutting MoE communication drastically.
Training foundation models for physics has always lagged behind training language models, and the reason is simple: the math is harder. A team from the Chinese Academy of Sciences just showed that with the right systems engineering, you can train an 11.5-billion parameter universal interatomic potential across an entire exascale supercomputer, hitting 1.2 EFLOPS at over 90% parallel efficiency. Their work introduces two things: MatRIS-MoE, a Mixture-of-Experts model for atomistic simulation, and Janus, the distributed training framework that makes it tractable.
Universal Machine Learning Interatomic Potentials (uMLIPs) are neural networks that predict the energy and forces of atomic systems with quantum-mechanical accuracy, but orders of magnitude faster than density functional theory (DFT). Think of them as foundation models for chemistry and materials science, trained on huge databases of DFT calculations spanning the whole periodic table.
Why training these models is uniquely painful
The bottleneck is not just scale. uMLIPs need something LLMs do not: force-matching via second-order derivatives, meaning training requires double-backward (second-order automatic differentiation), which doubles both computational costs and memory footprints from exploding intermediate activations. Forces are defined as the gradient of energy with respect to atomic positions, so computing a loss on forces means backpropagating through a gradient, which then has to be backpropagated again during optimization.
Two more constraints make it worse. Unlike LLMs that routinely exploit FP16 or FP8, uMLIPs must use full single-precision FP32 arithmetic to preserve the quantum-accurate fidelity essential for stable molecular dynamics. And the workload is enormous: the 473 million configurations used yield approximately 3.6 trillion interacting edges, since message passing along graph edges dominates compute.
Existing frameworks buckle under this. State-of-the-art frameworks optimized for first-order LLMs are fundamentally incompatible with the second-order computational graphs required for uMLIP force-matching, and this lack of native model parallelism effectively stalls the development of larger models. Meanwhile, model sizes have exploded: the 1.4B-parameter UMA model utilized 256 H200 GPUs for 21 continuous days.