MPs in PyTorch 2.15 (huggingface.co)

The PyTorch team at Hugging Face has released PyTorch 2.15, significantly enhancing the library's compatibility and performance on Mac systems with the Metal Performance Shaders (MPS) backend. This update marks a pivotal improvement for running and training models on Apple silicon, particularly with the introduction of float8 (FP8) tensors that effectively halve the memory footprint compared to BF16 and FS32 formats. Additionally, the update boasts a 5–6.5× increase in the decoding speed for mixture-of-experts models on M3 Ultra chips, driven by optimized operations that streamline GPU-CPU communication and enhance linear algebra performance on the GPU. Moreover, the update includes advancements across various domains, such as audio, video, and training operations, with many previously slow processes now streamlined, resulting in dramatic memory reductions—for instance, video decoding memory requirements plummeting from 270 GiB to just 8 GiB. Other technical enhancements encompass improved execution of convolutions, padding operations, and softmax calculations, which are now executed using dedicated Metal kernels, achieving an 8× performance boost. As the community anticipates these significant changes, the groundwork for PyTorch 2.16 is already underway, promising further optimizations and expanding the capabilities of AI/ML workloads on Mac systems.