Benchmark LLM Inference on WebGPU (arxiv.org)

A recent study has systematically explored the dispatch overhead of WebGPU during large language model (LLM) inference across various GPU vendors, backends, and browsers. This research evaluates performance for both small (0.5 billion) and larger (1.5 billion) models on four major GPU platforms: NVIDIA, AMD, Apple, and Intel, and reveals that naive benchmarking methods significantly overestimate the cost of WebGPU operations—potentially by a factor of 20. The findings highlight the actual per-dispatch overhead as being considerably impactful, with figures ranging from 24-71 µs, depending on the graphics API used. The significance of this work lies in identifying and characterizing the key performance bottlenecks in LLM inference enabled by WebGPU, emphasizing the critical role of backend and implementation choices. The study introduces the "torch-webgpu," a new PyTorch backend, achieving 11-12% of CUDA performance and demonstrating that kernel fusion can improve throughput by up to 53% on Vulkan. This research holds substantial implications for developers and researchers looking to optimize LLM workflows in web environments, as it provides a clearer understanding of how to maximize performance despite the inherent overhead introduced by WebGPU’s security-driven architecture. All findings and source code are available as open-source for further exploration and application.