Single CPU Language Modeling Benchmark Shattered (spasim.org)

The Hutter Prize for Lossless Compression of Human Knowledge has achieved its largest milestone in 20 years, with three entries significantly advancing the benchmark by nearly 10% each, all running on a single CPU core. Unlike traditional AI benchmarks that can be inflated by data leaks, the Hutter Prize evaluates models based solely on their ability to compress a publicly available 1GB snapshot of Wikipedia, thereby ensuring that any memorized data must be encoded within a smaller program, affecting the score directly. This unique format offers a more accurate measure of AI performance and intelligence. This notable progress is significant for the AI/ML community as it challenges the prevailing notion that advances in machine intelligence must correlate with increased data-center power consumption. By limiting participants to fixed compute resources, the prize promotes an environment where ingenuity and algorithmic innovation take precedence, allowing talented individuals beyond large labs to compete. With AI developments now leveraging superintelligence to enhance their algorithms, the competition is evolving, prompting the committee to adapt its judging process to keep pace with competing innovations. The next target for the prize remains at the forefront of advancing AI's capability while addressing societal concerns around resource consumption.