Why standard WER fails for Indian languages (www.sarvam.ai)

The recent discussion highlights the limitations of traditional metrics like Word Error Rate (WER) and Character Error Rate (CER) in evaluating Automatic Speech Recognition (ASR) systems for Indian languages. These metrics, initially developed for English, often fail to account for the fluidity of Indian languages, where colloquial and formal registers coexist, code-mixing is common, and words can have multiple valid representations. As a consequence, applying these metrics can misrepresent the true performance of Indic ASR systems, leading to misleadingly low accuracy scores. In response, the blog introduces a layered evaluation framework that incorporates advanced metrics such as LLM-WER and LLM-CER, which leverage large language models to assess meaning rather than mere character accuracy. This approach allows for a more nuanced evaluation, better reflecting how speakers perceive accuracy. The framework is exemplified through Saaras V3, a speech recognition API for 22 Indian languages that showcases various output modes, including transcription, translation, and code-mixing options. By adopting these new evaluation methods, developers can ensure that their ASR systems are effectively validated and optimized for the multilingual and dynamic nature of Indian speech.