The Nordic Token Tax (research.klang.ai)

A recent analysis revealed that large language models (LLMs) charges significantly more for processing non-English prompts, particularly for Scandinavian languages. Researchers examined the tokenization costs for the same set of 3,000 sentences in English, Swedish, Norwegian, Danish, and Finnish across twelve AI models, finding that all models charged between 30% and 100% extra for the latter languages. This increased cost is attributed to how tokenizers are designed, as the models read and bill text in tokens—short pieces of language—based on their vocabulary, which was predominantly built using English-language corpuses. This finding is crucial for the AI/ML community, highlighting an economic disparity in language processing that could hinder the adoption of AI tools in non-English-speaking regions. The study suggests a potential solution—developing local AI models with vocabularies tailored to specific languages could reduce costs. The results underscore the importance of advancing local AI labs in the Nordics and beyond, allowing for more equitable access to LLMs and fostering competition that could improve language processing efficiency for various linguistic demographics.