AI models cost more when you don't use English (restofworld.org)

A recent study by the Coalition for Language Equity in AI Research reveals that users of less commonly spoken languages may face a significant financial burden when utilizing AI models due to a hidden “tokenizer tax.” This phenomenon occurs because AI models require more tokens—data processing units—for languages with a lesser online presence. For instance, the study indicates that the most expensive languages can demand between 1.5 to 5 times more tokens than the cheapest, with disparities growing even larger when considering a broader set of 200 languages. Such inequities not only raise costs but also result in higher latency and reduced context capacity, complicating the delivery of quality AI services, particularly in areas with limited computing and connectivity resources. The implications of this study extend beyond mere costs, spotlighting the challenge of ensuring equitable access to AI technologies for non-English speakers. The current dominance of English-optimized tokenizers means that many languages, especially those with less digital representation or unique scripts, face significant disadvantages in processing efficiency. To address this tokenizer inequity, experts advocate for enhancing the underlying data quality and designing tokenizers that accommodate a broader range of languages and scripts. As AI continues to evolve, achieving a balance between cost-effectiveness and linguistic accessibility will be essential, leading to more inclusive participation in the AI economy.