Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks (skypilot.ai)

NVIDIA has unveiled its NVL72 systems, which significantly enhance the capabilities of AI and machine learning workloads by extending NVLink connectivity from a single node to an entire rack, allowing all 72 GPUs within a rack to communicate swiftly with unmatched bandwidth—totaling 130 TB/s. This advancement is crucial because traditional methods limit multi-node communication to slower interconnects like InfiniBand, leading to potential performance degradation of up to 43%. NVL72's architecture enables more efficient data exchange, which is essential for scaling distributed training jobs and mitigating communication bottlenecks associated with tensor and expert parallelism. The introduction of a topology-aware scheduling system by SkyPilot further optimizes GPU utilization by ensuring that workload slices are allocated within a single rack to preserve high-speed NVLink connections. This approach can increase usable capacity by 27% compared to traditional scheduling methods, while also minimizing the impact of GPU failures through in-place recovery without disrupting ongoing tasks. By improving both hardware efficiency and workload management, NVL72 systems promise substantial cost savings and performance improvements in large-scale AI applications, enabling researchers and organizations to maximize their computational resources and expedite machine learning processes.