Accelerating LLM Inference on AMD GPUs with Low-Latency GEMMs (rocm.blogs.amd.com)

AMD has announced a significant advancement in large language model (LLM) inference on their GPUs with the introduction of the LDS-Pipelined Split-K GEMM technique. This innovation aims to reduce decode-time latency, a crucial factor for user-facing applications like chatbots and coding assistants, by optimizing the GEMM (General Matrix Multiply) operations that form the backbone of LLM inference. The technique utilizes the on-chip Local Data Share (LDS) memory to efficiently manage the long K dimension across multiple computation threads, enhancing throughput and decreasing response times. The LDS-Pipelined Split-K method reveals a multi-layered approach by splitting K reductions across Compute Thread Arrays (CTAs) and using intra-CTA warps for parallel processing. Benchmarks indicate it achieves up to a 1.64x latency improvement over existing methods in decode operations. This improvement is particularly important as LLMs often deal with small M (active tokens) while processing high K (feature dimensions), leading to performance bottlenecks. By specifically targeting the unique shape requirements of LLM decode paths, AMD's innovation is poised to enhance the interactive capabilities of AI systems, significantly improving user experience and model serving efficiency.