ML Drift: Next-Gen GPU AI/ML Inference at the Edge (developers.googleblog.com)

Google's AI Edge Team has unveiled ML Drift, an open-source, cross-platform GPU compute engine designed for efficient on-device AI/ML inference. By abstracting the complexities of various GPU architectures and APIs such as OpenGL ES and Metal, ML Drift enables developers to create real-time, interactive machine learning applications across multiple platforms, from advanced video effects to generative AI. This engine serves as both the core GPU accelerator for LiteRT and a standalone library, ensuring peak performance even as AI workloads become increasingly demanding. The significance of ML Drift lies in its ability to address the challenges posed by hardware diversity at the edge, where traditional inference frameworks fall short. It introduces several key innovations, such as tensor virtualization for unified shader management, support for 5D tensors, and a modernized custom operation framework that simplifies adding specialized model components. These advancements not only elevate the performance of existing classical models but also facilitate the deployment of sophisticated generative architectures. With ML Drift already in production across millions of devices—including Google products like YouTube and Photos—it is poised to redefine the landscape of on-device AI, enhancing user experiences and pushing the boundaries of what mobile and edge devices can accomplish.