George Hotz's tinygrad Beats AMD's Best Llama 3.1 Training Time by 24 Seconds
tinygrad clocked a Llama 3.1 8B pretraining run on 2x8 MI350X in 108.5 minutes, edging past AMD's own MLPerf submission.
- tinygrad hit 108.5 minutes on Llama 3.1 8B pretraining across a 2x8 MI350X tinybox cluster.
- Beats AMD's top MLPerf Training v6.0 MI350X time of 108.9 minutes on the same benchmark.
- Run happened on a thermal-throttling machine cooled by fans, not air conditioning.
- Enabled by hierarchical allreduce and new multi-node MLPerf scripts merged upstream.
- Team is building a GPU assembly backend to match hand-written kernels LLVM cannot emit.
- Signals ROCm training software gap is narrower than vendor MLPerf tables imply.
George Hotz’s tinygrad project says it completed the MLPerf Llama 3.1 8B training workload in 108.5 minutes on two tinybox systems containing 16 AMD MI350X GPUs. That is 24 seconds faster than the best reported MLPerf Training v6.0 time on the same silicon.
| Run | Time | Status |
|---|---|---|
| tinygrad, 2 nodes and 16 GPUs | 108.5 minutes | Project-reported result |
| Best cited MLPerf v6.0 result | 108.9 minutes | Official benchmark result |
| AMD official submission | 109.76 minutes | Official benchmark result |
| AMD reproduction guide | About 110 minutes | Expected reproduction time |
The tinygrad number has not yet been identified as an audited MLPerf submission, so its leaderboard position depends on whether the run followed the same convergence target, timing rules, software constraints, and validation process. According to the project, the hardware also encountered thermal throttling because room cooling relied on fans rather than facility air conditioning.
The benchmark measures Llama 3.1 8B pretraining from randomly initialized weights on a prescribed subset of the C4 text dataset. AMD helped develop it as a more accessible counterpart to the Llama 3.1 405B workload while retaining features such as distributed training, mixed-precision computation, and synchronized gradient updates. The result provides a practical measure of mid-scale language-model training throughput.
A 24-second lead over the best listed run
AMD has historically supplied the prominent MLPerf results for its own accelerators, using an internally optimized software stack. Its Training v6.0 language-model submissions used the Primus framework for workloads including Llama 2 70B LoRA fine-tuning and Llama 3.1 8B pretraining.
tinygrad reached its 108.5-minute result through PR #18315, which adds the MLPerf workload to a tinybox_2x8xMI350X configuration. The implementation spans two machines with eight GPUs each, making communication between nodes a central performance constraint.
Hierarchical allreduce cuts synchronization costs
The supporting changes add hierarchical allreduce, two-node benchmark scripts, an ALLREDUCE_NODE_NDEVS context variable, and a unit-tested communication path. During distributed training, each GPU calculates part of the model’s gradient. Allreduce combines those partial results and returns the aggregate to every GPU before the next optimization step.
Hierarchical allreduce first combines data among GPUs within each server, where bandwidth is higher, then exchanges consolidated results between servers. That structure reduces traffic over the slower node-to-node connection and helps prevent communication from erasing the throughput gained by adding a second machine.
Handwritten kernels remain the next target
tinygrad says AMD’s reference stack still contains custom kernels that LLVM cannot generate, including routines written directly in GPU assembly. The project is building an assembly backend to express comparable low-level instructions and scheduling choices.
AMD’s recent training gains have drawn on MXFP4 recipes and specialized matrix-multiplication and attention kernels. These kernels control data movement, instruction selection, and use of the accelerator’s compute units. A general-purpose compiler can struggle to reproduce those choices, which explains tinygrad’s move toward an assembly path despite already matching the published end-to-end time.
What AMD developers can take from the result
AMD’s own evaluation places an eight-GPU MI355X system within roughly 5% of an eight-GPU Nvidia B200 system for Llama 2 70B fine-tuning and about 6% behind for Llama 3.1 8B pretraining. Training v6.0 also brought AMD’s first multi-node MLPerf submissions, extending its published results beyond single-server configurations.
The tinygrad run adds several practical signals for teams evaluating AMD training hardware:
- Primus is one route to competitive MI350X performance, while tinygrad now offers an open-source alternative with a comparable reported time.
- The compiler, runtime, distributed-training code, and benchmark integration are available for inspection and modification.
- A two-node ROCm configuration can scale effectively when gradient synchronization accounts for the network hierarchy.
- An assembly backend could improve performance further by targeting kernels that LLVM currently fails to emit efficiently.
For developers considering MI350X systems, the result shows that competitive training performance can come from an independent framework on a modest two-node cluster. Reproduced logs and an official MLPerf submission would establish direct comparability, but the current code already gives researchers a concrete implementation to inspect, run, and optimize.
Comments
