Network and Storage Benchmarks for LLM Training on the Cloud (maknee.github.io)

A recent benchmarking study highlights the critical impact of network and storage infrastructure on distributed large language model (LLM) training performance in the cloud—an area often overlooked in favor of model architecture and hyperparameter tuning. Using SkyPilot for infrastructure orchestration and Nebius GPU clusters, the author fine-tuned Gemma 3 12B and GPT-OSS-120B models under various configurations. The results reveal that deploying high-performance InfiniBand networking (400 Gbit/s) instead of standard 10 Gbit/s Ethernet yields a staggering 9-10x speedup in training throughput by drastically reducing communication overhead during gradient synchronization. Meanwhile, selecting optimal storage solutions dramatically influences data loading and checkpointing times, with local NVMe drives accelerating batch loading by up to 20x compared to object storage, and strategic use of SkyPilot’s cached S3 mounts balancing speed with durability. These findings underscore that GPU compute power is rarely the bottleneck in large-scale training; instead, data pipeline efficiency—driven by network and storage bandwidth—dictates overall GPU utilization and cost-effectiveness. The study further demonstrates how a combined infrastructure optimization, enabled by simple SkyPilot YAML flags, can shorten end-to-end training time by 6-7x. This work serves as a valuable guide for AI practitioners aiming to maximize cloud GPU investment and highlights the importance of tuning infrastructure alongside software for scalable, efficient LLM training.