Pretraining progress is mostly coming from data (www.dwarkesh.com)

Recent analysis reveals that the significant advancements in AI pretraining from 2019 to 2025 have predominantly stemmed from improvements in data rather than model architecture. The study, which evaluated various model recipes and data corpuses, found a staggering 3.24 times more compute efficiency gains attributed to data enhancements (12.0x) compared to model improvements (3.7x). This finding is crucial for understanding how future AI advancements will unfold, as it suggests that data engineering—focused on extraction and curation—plays a pivotal role in enhancing model performance, especially for smaller architectures. Moreover, the research indicates that the benefits of data and model improvements are mostly independent of each other, with additive effects observed in performance evaluations. While model improvements are essential for scaling up capabilities and managing training challenges, the results raise concerns about the sustainability of progress as potential limits on high-quality data availability may hinder the continued pace of innovation. As large models grow more robust, their performance may rely less on data quality, which complicates the landscape of AI research and development moving forward. Ultimately, this study highlights a critical intersection of data management and model innovation that will shape the economics and future trajectory of AI.