Scaling Unbounded N-Gram Language Models to a Trillion Tokens (infini-gram.io)

Researchers at the University of Washington and the Allen Institute for AI have unveiled Infini-gram, a groundbreaking n-gram language model capable of processing data on an unprecedented scale—trained on 5 trillion tokens and handling 5 quadrillion unique n-grams. This model redefines n-gram boundaries by allowing n to be arbitrarily large, enhancing the model's predictive capabilities significantly compared to traditional n-gram systems limited to values of 5 or less. With rapid query processing times of just 20 milliseconds for n-gram counting and a host of functionalities including estimating n-gram probabilities and document retrieval, Infini-gram presents new avenues for practical application in text analysis, language modeling, and more. The implications of Infini-gram extend deeply into the AI/ML community. By improving n-gram models, it complements existing neural language models and reduces perplexity by up to 73%, showcasing potential in applications like text generation, copyright protection, and data integrity verification. Additionally, it achieves this without the need for GPU resources, making it accessible for diverse computational environments. Infini-gram's efficient architecture, underpinned by a robust suffix array, signifies a shift towards managing vast text corpora more effectively, paving the way for innovative research and practical applications across various domains.