Show HN: TriusLD – A Small Tokenizer for LLMs with Robust Multilingual Support (trius.canton.graphics)

Trius has announced TriusLD, a new tokenizer designed specifically for large language models (LLMs) that supports 19 writing systems within a 65,535 token limit. This innovative tokenizer balances vocabulary to achieve robust multilingual compression, averaging 0.52 tokens per character across 58 languages, significantly outperforming GPT-4's 0.84 tokens per character. By maintaining a dedicated structure for each language script and integrating byte-level fallbacks, TriusLD enhances performance for languages traditionally underrepresented in AI models, like Hindi and Arabic, which often faced inefficient encoding challenges. This development is particularly significant for the AI/ML community as it addresses the long-standing issue of tokenization biases towards English and other dominant languages, helping models become more inclusive. TriusLD strategically allocates tokens to various writing systems—such as Cyrillic and Han—while preserving linguistic integrity and efficient character representation. By implementing a unique training methodology that segments language scripts, TriusLD ensures optimal merge decisions are made for each script, effectively reducing token costs and enhancing model comprehension across diverse languages. This tokenizer not only paves the way for more equitable AI applications but also strengthens multilingual capabilities in natural language processing tasks.