How OpenAI Delivers Low-Latency Voice AI for 900M Users (blog.bytebytego.com)

OpenAI has announced advancements in their voice AI technology, which now supports an impressive 900 million users weekly while maintaining low latency. To achieve this, they utilize WebRTC for live audio communication, but faced challenges with Kubernetes, given that its architecture treats server IPs and ports as disposable. Conventional solutions relied on Selective Forwarding Units (SFUs), suitable for multi-party calls, but OpenAI's heavy traffic is predominantly one-to-one interactions. Their new architecture overcomes this by separating the packet routing and protocol termination into a stateless relay and a stateful transceiver, optimizing for efficient resource management and lower latency. The key technical innovation involves using the ICE ufrag—a standard field exchanged during session setup—as a routing key, allowing the relay to determine the destination for initial packets efficiently. This enables OpenAI to facilitate rapid connection setups, ensuring that audio streams flow steadily and synchronously, thus preserving the conversational quality essential for voice AI applications. Additionally, the implementation of globally distributed relays further reduces latency by minimizing the distance data must travel, significantly enhancing the user experience. These developments mark a noteworthy step in making voice interactions feel natural and instantaneous, a critical requirement for maintaining user engagement in AI-driven interfaces.