Fast On-Device GenAI with LiteRT-LM (developers.googleblog.com)

Google AI has announced LiteRT-LM, a groundbreaking on-device generative AI solution designed to optimize the performance of the Gemma 4 model across a variety of platforms, including Chrome, ChromeOS, and iOS/Android apps. By leveraging advanced techniques such as Multi-Token Prediction (MTP) and optimized memory management, LiteRT-LM achieves up to a 2.2x speedup in inference, enabling real-time interactions without the typical latency of traditional models. This versatility allows developers to create applications that benefit from high-speed performance while minimizing resource consumption. The significance of LiteRT-LM for the AI/ML community lies in its ability to streamline the deployment of complex models on edge devices, offering low-latency, high-efficiency performance even within strict hardware constraints. Key features, such as advanced session management and seamless context preservation, enhance user experiences and reduce computational overhead. Additionally, its cross-platform capabilities extend to native support for iOS and web applications, providing a privacy-conscious alternative that empowers developers to build powerful, responsive applications effortlessly. With LiteRT-LM, Google positions itself as a leader in on-device AI, paving the way for innovative solutions in a wide array of use cases.