Interview with Richard Ho, OpenAI – By Dr. Ian Cutress (morethanmoore.substack.com)

OpenAI recently confirmed its foray into custom silicon development with the announcement of its inference accelerator, codename Jalapeño, designed in collaboration with Broadcom and manufactured by Celestica. This groundbreaking chip features 216 GiB of HBM4 with a peak performance close to 27 EFLOP/s when scaled to 2,048 accelerators, setting a new standard for power efficiency with up to 1.9 times better performance per watt compared to leading NVIDIA models. This innovation marks a significant shift in the AI/ML landscape as OpenAI moves to optimize its hardware for inference, contrasting with current industry practices that often rely on specialized chips for varying aspects of AI workloads. Notably, a substantial part of Jalapeño's design leveraged OpenAI's own advanced models, enabling an impressive optimization process that improved kernel efficiency dramatically. Richard Ho, the team's leader, emphasized the importance of a unified architecture that can seamlessly integrate into data centers, which deviates from the typical segmented approach adopted by competitors. By prioritizing a cohesive hardware strategy and fostering a collaborative environment filled with top talent, OpenAI aims to push the boundaries of AI performance while addressing existing limitations in memory and manufacturing capacities.