Retrofitting language models to operate over bytes (www.nature.com)

A recent advancement in AI, dubbed "byteification," enables retrofitting existing subword-based large language models (LLMs) to operate directly on byte encodings. This innovative approach addresses significant limitations of traditional subword tokenization, such as restricted character-level understanding and biases towards specific languages, especially in scientific contexts where minute details matter. Notably, the new byte-level models, including Bolmo 7B and Bwen 8B, achieve performance on par with or surpassing their subword counterparts while being trained with less than 1% of the pretraining resources generally needed for new models. The technical core of byteification involves a two-stage conversion process that repurposes existing subword models, leveraging their established architectures to create more efficient byte-level representations. The resultant models excel in tasks requiring complex character-level reasoning and maintain practical inference speeds. This breakthrough not only eliminates a long-standing performance gap associated with byte-level modeling, but it also allows for significant benefits in energy efficiency, reduces deployment costs, and enhances the understanding of nuanced textual content in specialized domains. By facilitating the swift adaptation of currently available LLMs to byte-level architectures, byteification paves the way for more advanced AI developments in diverse fields.