Compute Trends Across Three Eras of Machine Learning (2022) (arxiv.org)

A recent paper titled "Compute Trends Across Three Eras of Machine Learning" explores the evolution of compute requirements in machine learning (ML) and identifies three distinct eras: Pre Deep Learning, Deep Learning, and Large-Scale. The study highlights a significant shift in training compute growth; while compute doubled every 20 months before 2010, it accelerated to a doubling period of approximately six months post the rise of deep learning in the early 2010s. This trend further intensified around late 2015, with the emergence of large-scale ML models that require 10 to 100 times more compute resources. This research is crucial for the AI/ML community as it underscores the rapidly increasing computational demands necessary for advancing ML systems. By categorizing the historical context of compute into clear eras, the paper provides insights into how the underlying infrastructure and algorithms need to evolve to maintain pace with these burgeoning requirements. Understanding these trends is essential for researchers and organizations as they plan for future developments in ML technologies, highlighting both the challenges and opportunities that lie ahead in the field.