PyTorch 2.14 Ships Faster Apple Silicon, Rebuilt Distributed Training and NVGEMM

PyTorch 2.14 is out, built from 2,995 commits across 487 contributors since 2.13. The release concentrates on four areas: a new GEMM backend for NVIDIA GPUs, a rebuilt distributed communications stack, native linear algebra on Apple Silicon, and a unified way to declare dynamic shapes across compile and export.

NVGEMM lands as a real backend

The headline compiler addition is NVGEMM, a CuTeDSL-generated CUTLASS path inside Inductor. It first appeared in 2.13 as a way to emit standalone GEMM kernels, but anything downstream, a bias add, an activation, a rescale, still had to re-read the result from memory in a separate kernel. That gap closes in 2.14.

NVGEMM now uses NVIDIA's official cutlass.operators API, generating candidates that compete with Triton and ATen for mm, addmm, and scaled_mm. Kernels fuse epilogues the way Triton templates do: addmm's bias add, chained pointwise ops, and reductions over the GEMM result, including cases where the kernel returns both the reduced value and the full output matrix. Fusion reaches the low-precision paths as well, so pointwise work after a scaled GEMM folds into the kernel, and NVFP4's runtime global scale is applied inside the epilogue rather than as a separate multiply.

To enable it:

import torch._inductor.config as cfg
cfg.max_autotune = True
cfg.max_autotune_gemm_backends = "ATEN,TRITON,NVGEMM"

It requires nvidia-cutlass-dsl 4.6.0. The NVFP4 paths need Blackwell hardware, and anything the backend cannot express falls back to Triton.

A rebuilt distributed stack

The nccl2 backend, ported from the torchcomms experiment, implements the full Work contract on top of a reusable NcclApi abstraction with nonblocking communicators and eager communicator splitting. It is eager-only and adds one-sided windows, suspend/resume memory offload, and fault tolerance as first-class features. A nccl-lazy wrapper preserves the old lazy P2P initialization for workloads that need it.

Fault tolerance gets a conceptual overhaul. When a rank died in a large training job, the standard recovery was to tear down the process group and restart, discarding warm state everywhere. In 2.14, Backend and ProcessGroup expose reconfiguration interfaces so a group can be rebuilt in place, with abort hooks and pre/post collective hooks wired through the same path. Gloo picks up fault tolerance support alongside nccl2.

Three related additions round out the distributed changes:

• One-sided RMA windows. A rank can read or write peer memory without the peer posting a matching call, which suits irregular access patterns such as embedding lookups, weight transfer, and expert routing.
• Backend-agnostic Flight Recorder. The collective trace buffer was previously tied to NCCL. It now records through ProcessGroup hooks, so debugging a Gloo or custom-backend job no longer means losing the trace.
• Pluggable backends. Out-of-tree communications backends can register through Python entry points instead of patching c10d.

A new TokenSwitch module handles mixture-of-experts routing, with a TokenSwitchNCCL backend built on NCCL's expert-parallel kernels. It is early, private API requiring a build with USE_NCCL_EP=1 against the NCCL 2.30 pin, so for now it is NVIDIA-only.

Apple Silicon finally gets real linear algebra

MPS has long forced round-trips to CPU for anything past basic matmuls. SVD, eigh, and lstsq now run natively via Jacobi-style Metal kernels for float32 and complex64. Cholesky gets a faster panel-factorization algorithm with a matmul2d-based trailing update, roughly 1.2 to 2.8x faster depending on matrix size, plus a correctness fix for complex dtypes that previously had no dtype guard and could silently produce wrong results.

lu_factor and lu_solve move off Apple's MPSMatrixDecompositionLU to hand-written Metal kernels, with the contributor reporting over 100x speedups on small batched matrices. A new prefill attention kernel shows roughly 2 to 4x gains over the previous kernel across head dims and sequence lengths on macOS 26.2 and later. The release also includes a five-part reduction rewrite, a broad MPSGraph-to-Metal migration for common ops, and first-ever MPS forward and backward passes for CTC loss.

Anyone doing single-token decode on a Mac should note a specific fix: passing a [B, 1, K] activation to F.linear was falling off the fast path on MPS, costing an 8.5x slowdown on bf16 and fp16. The routing bug is fixed, and new GEMV kernels back vector-matrix shapes.

Dynamic shapes without the ceremony

torch.switch generalizes torch.cond to n-way branching without nesting, which matters most for mixture-of-experts models where nested conditionals bloated the traced graph. torch.while_loop can now be captured into a CUDA graph via CUDA's while conditional nodes, so a runtime-determined iteration count no longer forces a graph break.

The more significant addition is @dynamic_spec. Previously, declaring which dimensions vary required different mechanisms per entry point: a dynamic_shapes dict for torch.export, a coarse flag for torch.compile, a global mode for make_fx. A single decorator now covers all three:

from torch.fx.experimental.dynamic_spec import ShapeVar, dynamic_spec

@dynamic_spec(x=(ShapeVar("batch", min=2, max=128), None))
def forward(self, x):
    return self.linear(x)

Dimensions declared this way become unbacked symbols, so the compiler cannot quietly specialize on the batch size it happened to trace. Shape-dependent branching surfaces as a data-dependent error rather than a silent guard-and-recompile.

Platform and performance under the hood

Several changes deliver free performance for anyone already on torch.compile:

• Overlap on by default. Inductor's simple_overlap pass, which interleaves collectives with independent compute, is now opt-out rather than opt-in.
• Combo kernels handle large reductions. A single oversized reduction no longer determines the shape of an entire batched kernel.
• Compile-on-one-rank. Every rank in a distributed job previously compiled the same model independently. A new mode lets one process compile ahead of time and every rank load the same artifact, so a kernel compiled on cuda:0 loads and runs on cuda:3.
• AOTInductor zero-copy weight sharing. A new C API lets callers supply weight tensors at container creation, so multiple models can share one GPU-resident copy via CUDA IPC.

Platform coverage also expands: ROCm 7.14 wheels arrive via TheRock, Intel XPU gains native graph capture and symmetric memory for async tensor parallelism, and Inductor targets NVIDIA Rubin (sm_107) with tuned vectorized elementwise kernels. Python 3.15, including the free-threaded 3.15t build, is supported in eager mode. Calling torch.compile under Python 3.15 raises a RuntimeError immediately rather than falling back silently, so the limitation is visible from the start.

Worth upgrading?

Most 2.14 features carry an API Unstable designation, meaning interfaces can shift in subsequent releases. Production teams pinning versions should read the release notes carefully. Teams training MoE models, running distributed jobs on unreliable hardware, or shipping workloads on Apple Silicon will find several long-standing rough edges removed. Complex-valued tensors also gain experimental torch.compile support, decomposed into real and imaginary computations, opening the compiler to signal-processing and scientific workloads for the first time.

Install via the standard channel:

pip3 install torch --index-url https://download.pytorch.org/whl/cu130

The team is hosting a live Q&A with Andrey Talman, Natalia Gimelshein, Joe Spisak, and Chris Gottbrath. For pull-request-level detail, the GitHub release notes are the authoritative source.