Bolmo: Byteifying the Next Generation of Language Models (arxiv.org)

Researchers have announced Bolmo, the first family of fully open byte-level language models (LMs), featuring 1B and 7B parameters. Unlike traditional byte-level models that are trained from scratch, Bolmo utilizes a process called byteification to convert existing subword-level LMs into byte-level ones. This innovative approach addresses limitations of subword tokenization, such as inadequate character understanding and fixed vocabulary constraints, while maintaining performance that rivals leading subword models. Bolmo showcases impressive capabilities, surpassing existing byte-level LMs and even outshining the original subword LMs in tasks focused on character understanding and coding. It achieves competitive inference speeds by employing higher token compression ratios and presents a cost-effective solution for post-training by leveraging existing subword-level LMs. This development is significant for the AI/ML community as it positions byte-level models as a viable alternative to subword models, potentially broadening their application across various NLP tasks and enhancing the efficiency of language processing systems.