Current Large Audio Language Models largely transcribe rather than listen (arxiv.org)

Recent research highlights significant limitations in current Large Audio Language Models (LALMs) regarding their ability to comprehend emotional nuances in speech. A new benchmark called LISTEN (Lexical vs. Acoustic Speech Test for Emotion in Narratives) was introduced to assess how well these models differentiate between lexical and acoustic emotional cues. Analyzing six leading LALMs revealed a predominant reliance on lexical information, with models often misclassifying emotions as "neutral" when lexical cues are lacking. This raises concerns about their effectiveness in genuinely "listening" as opposed to merely "transcribing" spoken language, particularly in cases of cue conflict where emotion detection is critical. This finding is significant for the AI/ML community as it underscores the need for improvements in multimodal models that incorporate both acoustic and lexical analysis for better emotion understanding. By demonstrating that current models approach chance performance in paralinguistic contexts, the research suggests a pathway for developing more sophisticated systems that can truly interpret emotional intent in human speech. The LISTEN benchmark not only provides a clear framework for measuring progress in this area but also encourages further exploration of how LALMs can become more attuned to the complexities of human communication.