Running PyTorch Models on Apple Silicon GPUs with the ExecuTorch MLX Delegate (pytorch.org)

The newly announced ExecuTorch MLX Delegate enables optimized, GPU-accelerated inference for PyTorch models on Apple Silicon Macs, utilizing Apple's MLX framework. By seamlessly integrating with the PyTorch 2 export stack, the delegate supports a variety of quantization options (BF16, FP16, FP32, and more) and is compatible with diverse models, including dense transformers like Llama and speech-to-text systems like Whisper. The MLX delegate distinguishes itself by offering 3-6x higher throughput for generative AI workloads compared to previous ExecuTorch options, marking a significant performance enhancement for machine learning applications on macOS. This advancement is particularly noteworthy as Apple Silicon has gained traction as a powerful local platform for running large language models. The MLX delegate facilitates a straightforward workflow for developers, allowing easy model exports and execution while supporting around 90 essential ATen operations needed for transformer inference. With its emphasis on portability across various backends, the MLX delegate opens doors for more efficient real-time applications, from chatbots to transcription services, solidifying its role in the evolving landscape of AI/ML on Apple devices.