Turn 40: benchmarking voice agents over a long session (presence.bohita.com)

A recent study rigorously evaluated the performance of voice agents during long sessions—specifically examining the behavior by turn forty of a scripted conversation. The findings revealed significant gaps in key areas: latency, cost escalation, and engagement during cognitive tasks. Notably, while Gemini Live exhibited the fastest response times, it demonstrated a staggering seventeen-fold increase in cost from the first to the fortieth turn, due to failure to utilize cached context. In contrast, OpenAI's Realtime API maintained a more stable pricing structure throughout the session, leveraging caching to minimize cost variation. The study also highlighted an important factor often overlooked in single-turn benchmarks: real-time engagement during complex tasks. It found that while OpenAI can continue responding to user interactions during processing, Gemini Live remains silent, effectively freezing the conversation. This ability to maintain conversational flow even while accessing external resources positions OpenAI as a more user-friendly option for dynamic interactions. The implications of these findings are profound for the AI/ML community, underscoring the need for comprehensive benchmarks that reflect real-world usage scenarios, rather than focusing solely on isolated performance metrics.