Distributed Training of LLM's: A Survey (www.sciencedirect.com)

A new survey called "Distributed Training of LLMs" synthesizes current techniques and trade-offs for scaling large language models across many GPUs and nodes. It catalogs parallelism schemas (data, model/tensor, pipeline, and hybrid mixes), memory-saving methods (activation checkpointing, ZeRO/parameter sharding, FSDP), and communication strategies (All-Reduce, ring/recursive algorithms, compression and quantized gradients). The paper also reviews system-level optimizations — mixed precision (FP16/bf16 with loss scaling), CPU/NVMe offloading, memory disaggregation, and hardware considerations (NVLink, InfiniBand/RDMA, topology-aware placement) — plus software frameworks and libraries (Megatron-LM, DeepSpeed, FairScale) that make trillion-parameter training feasible. For the AI/ML community this matters because it consolidates practical recipes and measurement-backed guidance on where bottlenecks emerge (network bandwidth, optimizer state, and activation memory) and how choices affect throughput, convergence, and cost. The survey highlights emerging directions such as sparsity/MoE, communication compression, decentralized and federated regimes, checkpointing/fault tolerance, and autotuning for heterogeneous clusters. By clarifying technical trade-offs and open problems — energy and reproducibility concerns, programming complexity, and theoretical understanding of scaling interactions — it serves as a roadmap for researchers and engineers planning efficient, reliable LLM training.