LLM Evaluators are Biased across Languages (arxiv.org)

Recent research highlights significant biases in large language model (LLM) evaluators across different languages, suggesting that existing validation methods may not accurately reflect performance in multilingual contexts. The study analyzed semantically identical instruction-response pairs in 23 languages and discovered that evaluators produced notably different scores depending on the language used. This bias was particularly pronounced in lower-resource languages, which were scored more favorably, raising concerns about the reliability of LLMs in multilingual applications. Despite achieving over 90% pairwise accuracy on average, evaluators showed discrepancies of up to 43% in acceptance rates across languages, potentially allowing harmful content to bypass safety filters more easily in lower-resource scenarios. The findings underscore the need for a more nuanced approach to LLM evaluation that accounts for language-specific biases and variations. The researchers identified that model uncertainty plays a role in skewed scores, as models tend to give higher scores when uncertain. However, they also found that the bias persists even when controlling for uncertainty, indicating inherent structural misalignment at the language level. This raises critical questions for the AI/ML community regarding the development of fair and effective evaluation methodologies for multilingual models, as well as the implications for deploying AI in global contexts where language diversity is a significant factor.