With Nvidia Groq 3, the Era of AI Inference Is (Probably) Here (⌛ March 2026) (spectrum.ieee.org)

At this week's Nvidia GTC event, CEO Jensen Huang announced the launch of the Nvidia Groq 3 language processing unit (LPU), the company’s first chip explicitly designed for AI inference. This development comes after Nvidia’s significant $20 billion acquisition of Groq’s technology, positioning the Groq 3 as a frontrunner in the rapidly growing inference market. Huang emphasized that this marks a shift in AI capabilities, stating, "AI now has to think," underscoring the necessity for efficient inference in practical applications like chatbots and reasoning models. The Groq 3 LPU’s design diverges from traditional GPUs by utilizing integrated SRAM memory, which facilitates a linear data flow that reduces latency—crucial for real-time inference tasks. While it offers lower computational power compared to Nvidia's Rubin GPU, the Groq 3 boasts an impressive 150 TB/s memory bandwidth, significantly faster and more optimized for inference than conventional architecture. This innovation reflects a broader industry trend towards specialized chips for inference tasks, as evidenced by partnerships like those between AWS and Cerebras Systems. Nvidia's combined compute system, featuring both Groq 3 LPUs and Vera Rubin GPUs, aims to leverage the strengths of each chip, thereby enhancing overall AI performance.