CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion (arxiv.org)

A groundbreaking approach known as CommFuse has been introduced to tackle the pressing issue of tail latency during distributed training of large language models (LLMs). As LLMs expand in size, partitioning computational tasks across accelerators like GPUs and TPUs becomes necessary; however, this division often leads to substantial data communication overhead. CommFuse addresses this challenge by replacing traditional collective operations with a novel method utilizing decomposed peer-to-peer (P2P) communication, effectively facilitating finer-grained communication-computation overlap. This innovation promises to mitigate communication bottlenecks associated with tensor and data parallelism, potentially revolutionizing the efficiency of distributed training and inference. The significance of CommFuse lies in its ability to eliminate tail latency entirely, an issue previously seen in existing data slicing techniques. By providing an exact algorithm to reduce communication overhead, it enhances Model FLOPS Utilization (MFU) and increases throughput. The technique's versatility makes it compatible with various tensor-parallelism strategies, thus presenting a comprehensive solution for developers and researchers in the AI/ML community. Experimental results reveal consistent improvements in latency and performance, positioning CommFuse as a crucial advancement in the effective scaling of LLMs and distributed training processes.