Litert.js, Google's High Performance Web AI Inference (developers.googleblog.com)

Google has introduced LiteRT.js, a JavaScript binding of its LiteRT on-device inference library, enabling developers to run machine learning (ML) models directly in web browsers with enhanced performance. This innovation allows for complete local processing of models, ensuring improved user privacy, elimination of server costs, and low latency for real-time user experiences. With LiteRT.js, developers can seamlessly deploy existing TensorFlow Lite (.tflite) models to both mobile and desktop platforms, marking a significant upgrade from previous web solutions like TensorFlow.js, which utilized slower JavaScript-based kernels. The key technical advancements of LiteRT.js include support for advanced hardware acceleration using WebAssembly, leading to substantial performance enhancements—reportedly up to 3x faster than existing web runtimes across CPU and GPU. Notably, LiteRT.js leverages technologies such as XNNPACK for CPU acceleration, ML Drift for GPU, and the upcoming WebNN for neural processing units (NPUs). It simplifies the integration of AI capabilities in web apps, allowing functionalities like text generation and object detection to be handled client-side. In addition, it features streamlined model conversion processes from frameworks like PyTorch, empowering developers to optimize their models more effectively and rapidly implement high-performance, privacy-focused AI solutions in their applications.