Executorch: On-device AI across mobile, embedded and edge for PyTorch (github.com)

PyTorch has announced ExecuTorch, a groundbreaking solution for deploying AI models directly on devices ranging from smartphones to microcontrollers. This initiative, tailored for privacy and performance, will power on-device AI for Meta's platforms including Instagram and WhatsApp, enabling seamless integration of large language models (LLMs), vision, and speech applications. With native export capabilities, ExecuTorch eliminates the need for intermediate format conversions, enhancing workflow efficiency by allowing developers to deploy models with existing PyTorch APIs without rewriting code in C++ or facing vendor lock-in. Significantly, ExecuTorch employs ahead-of-time (AOT) compilation to optimize models for diverse hardware backends, boasting a lightweight 50KB runtime suitable for various devices. It supports over 12 hardware platforms including Apple, Qualcomm, and ARM, enabling developers to switch hardware targets with minimal changes in code. This versatility, along with built-in features for quantization and memory optimization, positions ExecuTorch as a valuable tool in the AI/ML community, facilitating rapid deployment and scaling of AI applications across devices while maintaining high performance and minimal resource use.