TurnBench (turnbench.sesame.com)

TurnBench has emerged as a valuable multi-domain benchmark designed to assess conversational turn-taking in dual-channel human dialogues. By meticulously annotating end-of-turn and interruption events across a 30-hour corpus of studio-recorded conversations featuring 154 dialogues and 106 actors, the benchmark provides a comprehensive evaluation framework. The leaderboard ranks models based on their recall for detecting these events, their false-positive rates, and latency, highlighting the trade-offs between accuracy and speed which are crucial for developing effective conversational AI systems. This initiative is significant for the AI/ML community as it addresses the complexities of real-time conversational dynamics, providing insights into the performance of various models like VAP, which excels in both end-of-turn and interruption tracking. The findings reveal that while human-like latency in turn-taking is approximately 151 ms before a turn ends, existing models struggle to balance high recall with low false-positive rates, especially in casual conversation contexts. Open-sourcing the scorer, baseline implementations, and offering interactive features on Hugging Face enhances accessibility, empowering researchers and developers to refine spoken interaction systems and advance the field of natural language processing.