Nvidia says Groq racks will be online this year following $20B purchase (www.cnbc.com)

Nvidia has officially launched its Groq 3 LPX chip into full production, following its monumental $20 billion acquisition of Groq, marking a significant step in the evolving landscape of AI/ML technology. This chip aims to enhance low-latency inference critical for AI applications, particularly in coding, allowing for faster response times and enabling cloud service providers to offer premium latency-sensitive service tiers. The new Groq 3 LPX racks, which will house 256 chips each and are expected to deliver an impressive throughput of 3,400 tokens per second, will be deployed at neocloud Nebius alongside Nvidia's Vera central processors and Rubin graphics processors. The significance of this development lies not only in the technological capabilities of the Groq chips, which include 500 megabytes of SRAM designed to mitigate memory bottlenecks, but also in the competitive dynamics it introduces. With companies like AMD integrating Cerebras chips for similar low-latency capabilities, combined with OpenAI's new Ultrafast mode, Nvidia's Groq offers a specialized solution that complements its powerful GPUs by addressing the decode phase of model serving. This strategic allocation of data center resources, as suggested by CEO Jensen Huang, indicates Nvidia's commitment to leveraging Groq chips for specific tasks while continuing to rely on GPUs for broader AI workloads.