A New Lens on Multi-GPU Comms: API-Level Memory Models for Portable Optimization (arxiv.org)

A recent study highlights the urgent need for optimization in multi-GPU communications, which are increasingly becoming a bottleneck in machine learning (ML) and high-performance computing (HPC) workloads. The paper identifies limitations in existing communication libraries like rocSHMEM and NCCL, which currently lack a comprehensive memory model to define how their functions interact with underlying memory systems. This deficiency can lead to brittle and error-prone implementations, undermining the potential of fine-grain optimizations available through low-level device APIs. To address these challenges, the researchers propose an API-level memory model that abstracts complexity for users, making it easier to implement optimizations without compromising performance. They present their model implementation within AMD's open-sourced rocSHMEM library, demonstrating substantial performance improvements—up to 2.48x for collective communications and even higher for compute + communication workloads. This approach not only enhances performance but also ensures a standardized interface, allowing developers to leverage advanced optimizations safely and effectively, marking a significant step forward in the pursuit of efficient multi-GPU systems in AI and ML applications.