Why speech-to-speech is the future for AI voice agents: Unpacking the AIEWF Eval (www.ultravox.ai)

The recently released AIEWF evaluation has highlighted Ultravox's speech-native model as a groundbreaking advancement in AI voice agents, outperforming traditional speech and text models. While standard evaluations like Big Bench Audio focus on a model's accuracy in understanding speech, the AIEWF eval shifts the paradigm by assessing real-world capabilities such as tool usage, instruction-following, and maintaining context across multi-turn conversations. Ultravox achieved impressive scores, including 97.7% overall accuracy and excellent latency performance, making it not only the top scorer in this evaluation category but also suitable for real-time applications. This shift towards speech-to-speech architectures, as demonstrated by Ultravox, promises to address key limitations of existing voice AI systems that rely on component stacks. By processing spoken input directly without the bottleneck of intermediate transcription, Ultravox reduces latency, minimizes error accumulation, and preserves the nuances of speech, leading to a more coherent conversational experience. As the demand for advanced voice agents grows in various applications, the success of Ultravox highlights the significance of robust evaluation frameworks tailored to real-world usage, setting a new benchmark for the future of voice AI.