PyTorch 2.15 Rebuilds Distributed Training and Cuts Decoding Overhead

PyTorch 2.15 combines 3,117 commits from 583 contributors, with most of the work concentrated in distributed training, low-precision decoding, and accelerator support. The release makes the rewritten NCCL backend the default, adds one-sided transfers through NIXL, gives FSDP2 finer mixed-precision controls, accelerates NVFP4 decoding on NVIDIA Blackwell GPUs, and brings FP8 operations to Apple Silicon.

Developers upgrading production systems should also account for three compatibility changes: Python 3.10 support has ended, C++20 is now required for native code, and distributed checkpoints created by PyTorch 2.15 cannot be loaded by earlier releases.

NCCL rewrite takes over

PyTorch uses communication backends to move tensors among processes during distributed training. Each process is called a rank, while coordinated operations such as all-reduce and all-gather are known as collectives. NVIDIA GPU jobs generally use NCCL for these operations.

PyTorch 2.15 completes the multi-release rewrite previewed in 2.14 and derived from the torchcomms implementation. Every process group created with the nccl backend now uses the rewritten path by default.

The most practical addition is a flight recorder that writes diagnostic state to disk when a collective hangs. These hangs often occur when one rank fails to reach an operation that every other rank is waiting to execute. The resulting post-mortem gives developers evidence for identifying the stalled rank and operation.

Applications can temporarily return to the previous implementation by setting TORCH_DIST_USE_NCCL2=0 or selecting the nccl-legacy backend. Both escape hatches are scheduled for removal in the next release, leaving one release cycle for migration and debugging.

The rewritten backend also adds:

• Runtime estimates for collective operations.
• An NCCL allocator for tensors owned by the backend.
• An opt-in reconfigure() path that can shrink or grow a process group.
• Error behavior aligned with the legacy backend.
• Window APIs and reconfiguration support on ROCm.

Dynamic reconfiguration is especially relevant to elastic training systems, which may need to continue after a worker or node leaves the job. Teams relying on that behavior should test recovery paths under realistic failures before enabling it in production.

NIXL opens a one-sided path

torch.distributed now includes a Python transport API for one-sided data movement, with NIXL as its first backend. A transport connects to a peer through a shared store and does not require a process group or a collective operation.

One-sided communication allows one rank to read from or write to a remote buffer without requiring the remote rank to issue a matching call at the same moment. That model suits disaggregated inference, KV-cache migration, parameter servers, and RLHF rollout systems whose producers and consumers operate on different schedules.

The NIXL adapter supports CPU and CUDA transfers. Its Transport.read_stream and Transport.write_stream methods enqueue transfers on a CUDA stream, allowing those operations to participate in graphs captured with torch.cuda.graph.

FSDP2 isolates sensitive weights

FSDP2 gains mixed-precision controls for parameters that lose stability at lower precision. Mixture-of-experts router gates and normalization weights can remain in FP32 while the rest of a model trains in BF16. Developers configure these exceptions through MixedPrecisionPolicy and param_dtype_override_fn.

The implementation retains one all-gather for each fully_shard unit. An all-gather reconstructs sharded parameters across ranks before computation, so avoiding additional calls reduces communication and synchronization overhead.

Other distributed-training changes include:

• CUDA Graph gradient accumulation: the microbatch loop can run as one replayed graph, reducing repeated CPU launch overhead.
• Overlapped gradient reduction: reduce-scatter communication can execute alongside backward computation.
• Persistent unsharded storage: FSDPModule.set_keep_unsharded_storage() keeps parameter storage allocated across reshard() calls, avoiding repeated buffer allocation.
• Pipeline accumulation: pipeline schedule step() accepts finalize_gradients, enabling gradient accumulation across multiple schedule runs with FSDP stages.

Distributed checkpoint metadata now records tensor strides. Checkpoints saved with PyTorch 2.15 therefore cannot be loaded by PyTorch 2.14 or earlier. Environments that require rollback should retain an older checkpoint or validate a separate conversion procedure before upgrading.

Blackwell decode sheds launch overhead

NVFP4 is NVIDIA Blackwell’s native 4-bit floating-point format. At equal element counts, 4-bit values require half the storage of FP8 values, while supported Tensor Core operations can offer higher peak throughput. Actual gains depend on scaling metadata, memory traffic, model shape, and kernel efficiency.

PyTorch 2.15 improves NVFP4 language-model decoding through NVGEMM, the CuTeDSL-generated CUTLASS kernel path introduced in 2.14. NVGEMM now uses fewer kernel launches and overlaps them more effectively. Small-batch decoding is often constrained by launch overhead, so reducing the number of launches can increase token throughput.

The cuDNN backend for torch.nn.attention.varlen also gains causal grouped-query attention with a paged KV cache. On Blackwell GPUs, the path supports an attention head dimension of 256 and requires cuDNN 9.24 or newer. Paged caches let serving systems manage KV memory in blocks instead of reserving one large contiguous region for every request.

Apple Silicon gains FP8 kernels

The Metal Performance Shaders backend adds float8_e4m3fn support on Apple Silicon. PyTorch 2.15 also implements _scaled_mm with tensorwise or rowwise scaling and provides a matrix-vector multiplication fast path for decode-sized shapes.

Mixture-of-experts workloads gain grouped_mm with a decode-specific fast path. PyTorch reports 1.32x to 1.34x higher MoE decode throughput on an M3 Ultra in its release benchmarks. Applications should repeat those measurements with their own expert counts, batch sizes, and quantization settings.

Public torch.mps.Stream APIs now allow applications to schedule concurrent work explicitly. The FP8 and decode paths require macOS 15 or newer.

Upgrade boundaries to check

Benchmark by workload

Multi-node training teams should first test NCCL collective performance, hang diagnostics, and failure recovery. Jobs using the temporary legacy fallback also need a migration plan before the next PyTorch release removes that path.

Blackwell inference deployments should benchmark NVFP4 decoding and paged-KV grouped-query attention with representative batch sizes, sequence lengths, and cache pressure. MoE training teams should validate FP32 overrides for router and normalization parameters, then measure whether the added stability affects memory use or throughput.

Disaggregated serving and RLHF systems can evaluate the NIXL transport API for direct buffer transfers between independently scheduled components. Apple Silicon applications should compare the new FP8 and grouped-matrix kernels on their target hardware, since the published M3 Ultra results may not transfer directly to other chips or model shapes.