PyTorch 2.9 released with C ABI and better multi-GPU support (pytorch.org)

PyTorch 2.9 is out with a broad set of upgrades aimed at making high-performance, multi-platform model development more portable and efficient. Highlights include a growing stable libtorch C ABI and C++ convenience wrappers (device utils, more torch::stable::Tensor APIs and stable ATen ops) to let third‑party C++/CUDA extensions be built against one torch version and run on another; Symmetric Memory for easy in‑kernel multi‑GPU programming over NVLink and RDMA (CUDA and NVSHMEM backends) with APIs exposed under torch.ops.symm_mem; and a new torch._dynamo.error_on_graph_break() toggle to control whether torch.compile errors or resumes on graph breaks. PyTorch also expands wheel variant support (ROCm, XPU, and CUDA 13), enables FlexAttention on Intel GPUs, adds Flash‑decoding based on FlexAttention for x86 CPU inductor (improving long‑context LLM decoding), and delivers Arm/AArch64 optimizations and broader CI coverage. For practitioners this means easier binary extension distribution, finer-grained multi‑GPU kernel patterns (one‑sided RDMA/IB‑GDA access, in‑kernel puts/gets, new collectives and nontraditional all_to_all variants for MoE), and more predictable compile behavior in mixed eager/compiled flows. Note many features are API‑unstable or experimental (symm memory, ABI surface, wheel variants), so expect ongoing iteration; but the release materially advances cross‑platform performance, custom kernel design, and packaging for heterogeneous hardware.