What Language Is This? Ask Your Tokenizer (arxiv.org)

A new language identification method, UniLID, has been introduced to enhance multilingual natural language processing pipelines. While current systems excel with high-resource languages, they often struggle with low-resource and similar languages. UniLID employs the UnigramLM tokenization algorithm, focusing on language-conditional unigram distributions within a shared tokenizer vocabulary. This innovative approach allows for efficient data usage, incremental language addition without retraining, and seamless integration into existing systems. The significance of UniLID lies in its ability to improve sample efficiency—in low-resource scenarios, it can achieve over 70% accuracy with just five labeled samples. Empirical evaluations demonstrate that it outperforms established benchmarks like fastText, GlotLID, and CLD3, particularly in fine-grained dialect identification. This advancement not only enhances language model performance in diverse contexts but also contributes to the ongoing development of more robust AI systems capable of handling the complexities of global languages.