CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure (arxiv.org)

CCL-Bench 1.0 has been introduced as a trace-based benchmarking toolkit designed to evaluate large language model (LLM) infrastructure more effectively than existing methods. Traditional benchmarks often provide limited insights, offering mere summary statistics that fail to clarify why certain configurations outperform others. In contrast, CCL-Bench records detailed execution traces along with YAML workload cards and launch scripts, creating a reusable repository of evidence for ML workloads. This toolkit enables the calculation of detailed metrics related to compute, memory, and communication efficiency, thereby addressing critical inefficiencies masked by superficial benchmarks. This development is significant for the AI/ML community as it reveals nuanced insights that can improve training efficiency. For instance, CCL-Bench demonstrates that higher compute-communication overlap can lead to longer training step times, implying potential flaws in parallelization strategies. It also highlights that enhancements in TPU interconnect bandwidth can deliver greater overall performance improvements compared to similar boosts in GPU bandwidth, especially for smaller workloads. Furthermore, it shows that the optimal configuration for one training framework can be significantly slower than the best configuration on another framework, illustrating the importance of context-dependent tuning in model training.