PyTorch 2.13 is out, and it's a release that pushes hard in three directions at once: hardware breadth (Apple Silicon, Arm, AMD, Intel), memory efficiency for large-vocabulary LLM training, and distributed training at scale. The release is composed of 3,328 commits from 526 contributors since PyTorch 2.12. That's a lot of surface area, so here's what actually matters.
FlexAttention finally lands on Apple Silicon
FlexAttention is PyTorch's unified API for expressing custom attention patterns -- things like sliding window attention, document masking, or ALiBi -- as plain Python functions that get compiled into fused kernels. Before 2.13, you needed CUDA to use it. Now it runs on Metal/MPS (Apple's GPU compute stack).
The MPS implementation provides hand-written Metal kernels for both the sparse prefill and decode paths, including GQA and captured buffers. Instead of writing a custom CUDA kernel for every attention variant, FlexAttention lets you write a two-line Python function and the compiler builds a fast kernel automatically.
The benchmark numbers for sparse patterns are striking. On long, sparse attention patterns, the speedups over SDPA are substantial -- on a 1x8x32768x64 shape with a 256-element sliding window (0.8% density), FlexAttention runs in ~35 ms vs ~431 ms for SDPA, a ~12.3x speedup; a smaller 8192-length / 64-window case achieves ~4.15x. Dense patterns still favor SDPA, as expected.
There's also a correctness fix for CUDA users: by default, the FlexAttention flash backend uses atomic operations in the backward pass for dQ accumulation, which introduces non-determinism -- repeated runs on the same input can produce slightly different gradients. The new deterministic backward path replaces atomics with a pre-computed write ordering that guarantees bit-for-bit reproducible gradients. The measured end-to-end overhead is well under 1% at longer sequence lengths -- just +0.2% at S=32768 -- making determinism effectively free for most production workloads. You opt in via the existing
Don't miss what's next in AI
Join 300,000+ engineers and researchers who get the signal, not the noise.
- Full access to in-depth AI research breakdowns
- Be the first to know what's trending before it hits mainstream
- Daily curated papers, repos, and industry moves