Skip to main content
Back

GPU MODE's Free 100-Episode Series Closes the Gap Between PyTorch and Real Hardware

GPU MODE's open lecture series and GitHub repo bring real-world GPU kernel expertise to anyone willing to learn, for free

Repo, Full Access
  • GPU MODE is a free, community-run lecture series on GPU programming with 100+ episodes, a GitHub repo with 6,200+ stars, and a 28,000-member Discord.
  • Created by Mark Saroufim and Andreas Köpf, it started as a reading group for the PMPP textbook and grew into a full curriculum.
  • Lectures cover the full GPU stack: CUDA kernels, Triton, Flash Attention, quantization, multi-GPU communication, CUTLASS, SASS, and more.
  • Speakers include Jeremy Howard, Adam Paszke, Meta PyTorch engineers, and contributors to FlashInfer, Liger Kernel, and Unsloth.
  • The ecosystem includes a LeetGPU-style kernel leaderboard and GPU Puzzles for hands-on practice from day one.
  • All content is free; videos on YouTube, code and slides on GitHub, community on Discord at discord.gg/gpumode.

There's a gap between knowing PyTorch and knowing what PyTorch is actually doing on the hardware. Most ML practitioners live comfortably on the Python side of that gap. GPU MODE is a community-run initiative that wants to close it, and their free lecture series, now past 100 episodes, has become one of the most comprehensive open resources for GPU programming in the ML ecosystem.

From reading group to 28,000-member community

GPU MODE is a GPU reading group and community created by Mark Saroufim and Andreas Köpf. What started as a structured read-through of the classic textbook Programming Massively Parallel Processors (PMPP) has grown into something much larger. The GPU MODE Discord now has over 28,000 members, and the community-driven lecture series spans topics from profiling to kernels to CUTLASS to SASS (SASS is the low-level assembly that runs natively on NVIDIA GPUs).

Recordings of the weekly lectures are published on the YouTube channel, and material including code and slides for individual lectures can be found in the lectures repository. The GitHub repo itself has crossed 6,200 stars with over 620 forks, a signal that people aren't just watching but actually working through the material.

What's actually in the curriculum

The lecture list reads like a tour of the entire GPU software stack. Early lectures ease you in with PyTorch integration and PMPP chapters. Then the series goes deep fast. A sampling of what's covered: