Towards Compute-Aware In-Switch Computing for LLMs on Multi-GPU Systems (arxiv.org)

A new framework called CAIS (Compute-Aware In-Switch Computing) has been introduced to enhance tensor parallelism in large-scale LLMs (Large Language Models) on multi-GPU systems. Traditional approaches often suffer from inefficiencies due to frequent collective operations that impede inter-GPU communication. CAIS addresses this by synchronizing communication modes with the memory semantics essential for LLM computations, thereby reducing resource underutilization. The framework incorporates three key innovations: a compute-aware ISA and microarchitecture extension, a merging-aware thread block coordination for better request management, and a graph-level dataflow optimizer to enhance cross-kernel overlap. The significance of CAIS lies in its ability to greatly improve training speed, achieving up to 1.38 times faster end-to-end training compared to existing NVLink SHARP (NVLS) solutions, and 1.61 times faster than the top competing compute-communicate overlap methods. This advancement in structure offers a promising step forward for the AI/ML community, allowing for more efficient training of LLMs on multi-GPU setups, ultimately facilitating faster advancements in AI research and applications.