LiteRT – Google's Edge ML Framework (github.com)

Google has launched LiteRT, an advanced edge machine learning framework designed for high-performance on-device deployment of ML and Generative AI models. Building on the legacy of TensorFlow Lite, LiteRT simplifies the integration of on-device inference with features such as automated accelerator selection, true asynchronous execution, and efficient I/O buffer handling, making it easier for developers to optimize their applications for a variety of hardware. The framework now includes unified access to NPUs from major chipset providers, significantly enhancing GPU performance with zero-copy interoperability, which minimizes latency for multi-platform support across Android, iOS, Linux, macOS, Windows, and IoT devices. The significance of LiteRT lies in its potential to improve the efficiency and accessibility of deploying complex machine learning models at the edge, fostering innovation in mobile and embedded applications. LiteRT's new Compiled Model API offers tools for both beginners and experienced developers to easily run pre-trained models and expedite the deployment of large language models or diffusion models. With a commitment to expanding hardware acceleration and improving developer tools, Google positions LiteRT as a key player in the future landscape of on-device AI, inviting contributions to further enhance its capabilities.