New Prefill Specialised GPU – Nvidia Rubin CPX (semianalysis.com)

Nvidia has unveiled the Rubin CPX, a specialized GPU designed to optimize the prefill phase of AI inference by prioritizing compute throughput over memory bandwidth—a shift that marks a significant evolution in hardware architecture for large-scale AI serving. Unlike traditional GPUs that rely heavily on expensive high-bandwidth memory (HBM), Rubin CPX uses more cost-effective GDDR7 memory with lower bandwidth (2TB/s versus 20.5TB/s on Nvidia's R200) but delivers 20 PFLOPS of dense FP4 compute. This realignment addresses the imbalance in resource demands during the inference prefill stage, where compute dominates but high memory bandwidth is underutilized, enabling more efficient hardware specialization and reducing the total cost of ownership. Technically, Rubin CPX is a monolithic chip featuring 128GB of GDDR7, eliminating the need for complex HBM packaging and NVLink interconnects, instead relying on PCIe Gen 6 for network scale-out. This design is integrated into Nvidia’s upgraded Vera Rubin (VR) rack architecture, offering three configurations that combine Rubin CPX and R200 GPUs to maximize performance and efficiency. The new rack solutions achieve a staggering system memory bandwidth of 1.7 PB/s from a blend of 144 CPX and 72 R200 GPUs, with liquid cooling to manage a higher power budget. Nvidia’s introduction of Rubin CPX not only widens the gap with competitors like AMD and ASIC providers but also sets a new industry standard for disaggregated inference serving, pushing others to rethink their silicon roadmaps and software stacks in response.