George Hotz's Tinygrad Crashes MLPerf With 25,000 Lines of Python

Tinygrad, the open-source deep learning framework built by George Hotz and a six-person team, just made its debut on the official MLPerf Training v6.0 leaderboard. The submission: Llama 3.1 8B pretraining on AMD MI350X GPUs, using a completely custom driver, runtime, kernel library, and training loop. No PyTorch. No ROCm. No CUDA. Just ~25,000 lines of Python.

What MLPerf is and why it matters

The MLPerf Training benchmark suite measures how fast systems can train models to a target quality metric. Think of it as the F1 circuit for AI hardware and software: every major player submits their best setup, results are peer-reviewed, and the numbers are public. The MLPerf Training benchmark suite comprises full system tests that stress models, software, and hardware for a range of ML applications, and the open-source, peer-reviewed suite provides a level playing field that drives innovation, performance, and energy efficiency for the entire industry.

MLCommons announced new results for the MLPerf Training v6.0 benchmark suite today, with two new benchmarks added in this round highlighting rapid and significant changes in the AI ecosystem. NVIDIA delivered a clean sweep in MLPerf Training v6.0, the latest edition of industry-standard AI training benchmarks developed by the MLCommons consortium. But the more interesting story is who else showed up: MLPerf Training v5.0 marked AMD's first-ever training submission , and now tinycorp is on the board too, listed alongside AMD, Google, Azure, CoreWeave, and NVIDIA as one of 24 submitting organizations.

A 25,000-line stack that shouldn't be able to do this

The conventional wisdom in AI infrastructure is that you need millions of lines of battle-hardened software to run frontier training workloads. CUDA alone took NVIDIA nearly two decades to build. NVIDIA's CUDA ecosystem encompasses roughly 5.9 million developers, 18 years of accumulated libraries including cuDNN, cuBLAS, TensorRT, NCCL, and CUTLASS, and first-class integration with every major ML framework. Tinygrad's answer to all of that is a codebase that is, as Hotz put it, "1000x smaller."

Tinygrad is an open-source deep learning framework designed for simplicity and extensibility, featuring a PyTorch-like frontend for tensor operations, automatic differentiation via autograd, and a compiler that generates hardware-specific kernels. It was developed by George Hotz and first released in 2020 as a lightweight alternative inspired by the educational simplicity of micrograd and the ergonomic API of PyTorch.

The key architectural bet tinygrad makes is radical laziness. Every Tensor operation in tinygrad is lazy: calling a + b or x.relu() does not execute anything immediately. It appends a node to an in-memory Directed Acyclic Graph (DAG), and computation only runs when a result is explicitly requested, typically via Tensor.realize() or by converting to a NumPy array. This deferred execution lets the compiler see the full computation graph before generating any code, enabling aggressive operation fusion that eliminates redundant memory reads and writes.

Tinygrad compiles a custom kernel for every operation, allowing extreme shape specialization. All tensors are lazy, so it can aggressively fuse operations. The backend is 10x+ simpler, meaning optimizing one kernel makes everything fast. The entire model is decomposed into just three primitive operation types: elementwise ops (add, multiply, relu), reduction ops (sum, max), and movement ops (reshape, permute). Everything else, including convolutions and matrix multiplications, is expressed through these primitives and then fused by the compiler.

The full-stack sovereignty play

What makes the MLPerf submission remarkable is not just that tinygrad trained Llama 8B. It's that tinycorp wrote every layer of the software themselves. The tweet explicitly calls out: their driver, their runtime, their kernels, and their training loop. To fund the operation, they have a computer sales division that makes about $2M revenue a year, and they have a contract with AMD to get MI350X on MLPerf for Llama 405B training.

Hotz has been building toward this for years. He believes NVIDIA's CUDA moat is not insurmountable, and developed the open-source deep learning stack tinygrad with approximately 20,000 lines of code to challenge the CUDA ecosystem from the software side. The AMD MI350X is a key part of that strategy. MI350 Series GPUs have native support for FP4 precision, offering up to 20 petaflops of FP4 performance, combined with an industry-leading 288 GB of high-bandwidth HBM3e memory with a bandwidth of 8 TB/s.

The current tinygrad result on Llama 8B sits at 170 minutes. That's not the fastest time on the board, but that's not the point. The point is that it's on the board at all, submitted by a six-person team with no proprietary libraries, running on AMD hardware that the broader ecosystem has historically struggled to support for training workloads. AMD's MI350X matches B200 on FP8 compute and exceeds it on memory, but NVIDIA's software maturity delivers 50-55% MFU vs AMD's ~45%, meaning real-world performance per dollar favors NVIDIA for training. Tinygrad is directly attacking that software gap.

What comes next

Tinycorp has already announced the next milestone: Llama 405B on the next MLPerf round, along with an improved time on the 8B benchmark. They signed a contract with AMD to use the MI350X for training Llama 405B on MLPerf, and the negotiation process for this contract was largely conducted publicly on Twitter. That's a 50x scale-up in model size, and it would be a genuinely significant proof point if tinygrad can pull it off.

The MLPerf Working Group co-chair noted: "We're seeing strong convergence on a set of best practices for training AI models, but at the same time there is increasing technical diversity in the underlying frameworks and systems that are being used to host and run them." Tinygrad is exhibit A for that diversity. Tinybox previously competed against machines costing approximately 10 times more in the MLPerf Training 4.0 benchmark.

Why this changes the conversation

The assumption baked into AI infrastructure pricing is that the software stack is irreplaceable, which justifies enormous margins on cloud GPU compute. Hotz's argument, now backed by a peer-reviewed benchmark submission, is that this is not structurally true. A small team with a clean codebase can replicate what took hyperscalers years and billions of dollars to build.

Here is what tinygrad being on the MLPerf board actually unlocks:

• A credible alternative software path for AMD hardware. AMD GPUs have the memory and compute, but have historically lacked the software maturity. Tinygrad demonstrates a path to closing that gap without depending on ROCm reaching PyTorch-level polish.
• A reference implementation for sovereign AI stacks. Any organization that wants to run training without depending on NVIDIA or Google's software ecosystem now has a working, auditable example under 25,000 lines.
• Competitive pressure on cloud training pricing. If a six-person team can hit MLPerf-grade training performance, the argument that frontier training requires hyperscaler infrastructure gets harder to sustain.
• A template for hardware startups. The tinygrad approach, build the full software stack first before touching silicon, is a direct rebuttal to the chip-first strategy that has burned many AI hardware companies.

The 170-minute Llama 8B time will improve. The 405B submission is coming. And the tinygrad codebase remains fully open-source, meaning anyone can read exactly how a frontier training run works, end to end, without a single proprietary dependency in sight.