CAD: Disaggregating Core Attention for Efficient Long-Context LLM Training (hao-ai-lab.github.io)

Recent advancements in long-context large language model (LLM) training have highlighted the issue of workload imbalance, particularly in how core-attention computations are colocated with other model components. This imbalance leads to significant slowdowns, especially as context length increases, with reports of training inefficiencies scaling from 1.44x to as high as 4x. The new technique, Core-Attention Disaggregation (CAD), proposes separating the core-attention computation from linear components, allowing for more efficient utilization across GPUs. By treating core-attention as an independent task, CAD enables better balancing of workloads and significantly enhances the scaling of LLM training. The authors introduced a prototype system called DistCA, demonstrating up to 1.35x speedup over existing training systems. Notably, CAD reduces stragglers and pipeline bubbles often caused by uneven workloads. The disaggregation addresses memory and communication inefficiencies through innovative methods that minimize overhead while leveraging the specific characteristics of core-attention computations. The implications of this work are profound, potentially easing the high costs and resource demands associated with training long-context LLMs, thus providing a more scalable approach essential for developing next-generation AI applications.