Don't Move the Data (parallelprogrammer.substack.com)

This updated 2017 essay argues a simple but urgent point for modern AI systems: raw GPU compute has far outpaced the ability to feed it with memory and interconnect bandwidth, so the dominant performance constraint is moving data, not FLOPS. Chip performance rose from G80’s 0.38 TFLOPS (2006) to multi‑tens of TFLOPS and now to Blackwell-era chips with ~104B transistors and HBM bandwidths quoted around 8,000 GB/s per die, while commodity PCIe interconnects remain two orders of magnitude slower (PCIe 6.0 ~128 GB/s unidirectional). HBM (P100 onward), NVLink, GPUDirect and TensorCores helped — and deep learning serendipitously fits GPUs well because model weights can reside on device — but the FLOPS/byte gap still drives architectural and software trade-offs. For the AI/ML community this means practical and strategic guidance: “Don’t move the data.” Optimize by keeping data on-device (operator fusion, JITs like Numba), use coherent high‑bandwidth fabrics (NVLink/InfiniBand, GPUDirect), and prefer packaging and memory architectures that minimize I/O. System designers must rethink memory hierarchies, on‑chip SRAM/HBM capacity, and cache‑coherent interconnects to avoid starvation; software teams should prioritize data locality and reduced passes over tensors. The piece also traces NVIDIA’s responses (Mellanox, Grace, NVLink) and market implications — showing that solving data movement is now the central engineering and business challenge for next‑generation AI hardware.