What Nvidia's first Groq 3 LPU benchmarks tell us about its $20B gamble (www.theregister.com)

Nvidia has unveiled impressive benchmarks for its Groq 3-based LPX racks, achieving a staggering 3,400 tokens per second (tok/s) processing speed with a 100,000-token input sequence on Google’s Gemma 4 31B model. This marks a fourfold speed increase over competitor Cerebras' performance, underlining Nvidia's strategic $20 billion investment in Groq's LPU technology. The Groq 3 LPUs utilize a unique SRAM-heavy architecture, offering memory bandwidth of 150 TB/s, which enables faster inference by addressing the memory bottleneck traditional GPUs face. The significance of this development lies in its potential to enhance the AI application's performance, particularly in generating longer and more complex responses in real-time, thereby improving user interaction. While the benchmarks reflect optimal conditions, challenges remain for scaling the architecture with larger models, such as mixture of experts (MoE) models that require even greater resources. Nvidia anticipates that combining its GPUs with Groq's LPUs will foster a heterogeneous inference system, optimizing performance across multiple concurrent users while catering to both the compute-intensive and memory-heavy phases of model inferencing. However, comparisons with Cerebras may need reevaluation as it rolls out next-generation accelerators, suggesting that the competitive landscape in AI inferencing will continue to evolve rapidly.