Why AI works worse in your language (artifipedia.com)

Recent analysis reveals significant disparities in AI performance across languages, primarily not due to linguistic complexity but rather due to imbalances in training data, tokenizer efficiency, and the scope of instruction-tuning. AI models trained predominantly on English demonstrate surprising multilingual capabilities, yet they show substantial performance gaps when handling non-English languages. For instance, accuracy on enterprise retrieval tasks sees drops of up to 29% when switching from English to other languages, and models can perform as poorly as 38% on indigenous languages where training data is scarce, often resembling random guessing. The findings highlight that performance is tied closely to the availability of training data and how well the tokenizer can handle various scripts. Models suffer from inflated processing costs and degraded performance due to inefficiencies in tokenization—particularly for non-Latin scripts and agglutinative languages. The research underscores the need for a more balanced digital representation of all languages on the internet, as well as improvements in tokenizer designs, to enhance AI functionality across diverse linguistic contexts. Moreover, the study points to a potential utility in prompting models in English for better performance in output requests for target languages, suggesting a complex interplay in cross-lingual understanding that warrants further exploration.