Sharded matrices and how to multiply them (jax-ml.github.io)

Recent developments in large-scale machine learning (ML) training have introduced a novel approach to sharded matrix multiplication, focusing on effectively managing and distributing the computation across multiple TPUs or GPUs. The core of this work lies in understanding how to multiply matrices that have been segmented, or "sharded," across different devices—a common necessity when model parameters exceed a single device’s capacity. Through a simplified theoretical framework based on TPU communication primitives, researchers have elucidated how to retain the logical shape of matrices while utilizing a device-specific view to facilitate efficient computations. This advancement is particularly significant for the AI/ML community as it allows for optimized use of distributed hardware resources, enhancing training efficiency for large language models (LLMs) and other extensive ML applications. By employing a systematic naming and partitioning approach for tensor dimensions based on device arrangements, the research not only clarifies the local shapes of sharded arrays but also highlights optimal communication strategies during matrix multiplication. The findings suggest that when merging sharded data, intelligently accounting for the arrangement of contracted dimensions can significantly reduce communication overheads, thereby improving performance in large-scale ML tasks.