GEMM Performance Optimization Across Generations of Ryzen AI NPUs (arxiv.org)

A recent study has highlighted significant performance optimizations for general matrix multiplication (GEMM) algorithms tailored for AMD's Ryzen AI XDNA and XDNA2 Neural Processing Units (NPUs). As deep learning workloads become increasingly demanding, the need for specialized hardware has grown, making this optimization crucial. By employing a systematic approach that leverages the unique architectural features of both NPU generations, researchers achieved impressive performance metrics: up to 6.76 trillion operations per second (TOPS) in 8-bit integer precision for XDNA and 38.05 TOPS for XDNA2, alongside 3.14 TOPS and 14.71 TOPS in brain floating-point (bf16) precision, respectively. This work is significant for the AI/ML community as it not only enhances the efficiency of hardware used in deploying deep learning applications but also provides insights into optimizing computational workloads on AMD's NPUs. Such advancements may lead to faster training times and more efficient inference, ultimately benefiting a variety of AI applications across cloud and edge scenarios, from natural language processing to image recognition. The findings pave the way for further innovations in hardware architecture, driving the evolution of AI capabilities.