Large Text Compression Benchmark (mattmahoney.net)

A new benchmark for evaluating lossless data compression algorithms has been announced, focusing on a dataset comprised of the first 109 bytes of an XML text dump of the English version of Wikipedia from 2006. Hosted by Matt Mahoney, this open benchmark seeks to advance research in artificial intelligence and natural language processing (NLP) by emphasizing the importance of modeling in distinguishing between high and low probability text sequences. Participants can contribute their results under specific guidelines, with potential eligibility for the Hutter Prize for innovative compression methodologies. The benchmark ranks various compression programs based on their performance, specifically the combined size of the compressed data and the decompression software. As participants optimize their algorithms for maximum compression—even at the expense of speed and memory usage—the rankings provide valuable insights into state-of-the-art techniques. Notable entries include variants of traditional methods like LSTMs and emerging neural network approaches, such as the fx2-cmix-transformer, highlighting the blend of classical and contemporary strategies in the pursuit of more efficient data handling—a key concern for both the AI/ML community and broader computing applications.