Layer Normalization as Fast as Possible (fleetwood.dev)

A new exploration of implementing Layer Normalization (LayerNorm) in WebGPU has led to significant advancements in performance and numerical stability, crucial for handling large neural network models like LLaMA and Whisper. As deep learning increasingly relies on LayerNorm over traditional batch normalization for stabilizing inputs, efficient implementations become vital, especially for running models in browsers with GPU support. The article details the development of a fast LayerNorm kernel by employing Welford's algorithm, which allows for precise calculations of mean and variance with minimal performance overhead. The exploration illustrates the challenges of GPU programming, particularly the hurdles of catastrophic cancellation when calculating variances. By leveraging Welford's online algorithm and the new WebGPU subgroup proposals, the implementation achieves performance levels comparable to one-pass methods while maintaining numerical accuracy. This approach not only enhances computational efficiency but also minimizes common precision issues, showing promising implications for the future of real-time, browser-based deep learning applications.