OpenAI Jalapeño: Better than Nvidia Blackwell (newsletter.semianalysis.com)

OpenAI has officially unveiled its latest inference chip, “Jalapeño,” at the Hot Chips conference, claiming it outperforms Nvidia's Blackwell across various benchmarks. Developed in partnership with Broadcom over an impressive timeline of around 16 months, Jalapeño is designed from scratch specifically for large language model (LLM) inference but is touted as a generalized chip capable of handling multiple AI workloads effectively. Early performance metrics show that Jalapeño excels in both low-latency and high-throughput contexts, achieving remarkable token throughput per watt without the need for multi-token prediction, making it a significant advancement in chip design for AI applications. The architecture of Jalapeño includes HBM4 memory, providing an impressive 15.4TB/s bandwidth, surpassing current accelerators utilizing HBM3E. This positions Jalapeño competitively against Nvidia's and AMD's top-tier offerings, while its performance per watt aligns with OpenAI's focus on energy efficiency within power-constrained datacenters. Notably, the chip’s design philosophy opts for a homogenous pool of resources that adapt to varying workload demands, rather than over-specializing for specific tasks. This strategic approach, combined with robust benchmarking results from the InferenceX suite, underscores Jalapeño's potential to reshape the landscape of AI inference hardware.