A Stargate for Data: What do we do after we run out of internet to train on? (twitter.com)

Labs are projected to spend over $100 billion annually on data by 2030, as the AI and machine learning community grapples with a critical scarcity of high-quality data. As deep neural networks show consistent improvement with increased scaling of both model size and training data, the trend is shifting toward a data-limited regime. The Internet, which has historically provided a wealth of information, is no longer sufficient to meet the exponential demand for training data. With only about 300 trillion tokens of useful public human text available, labs must now invest in private datasets and human-generated data to fill the gaps. This shift is significant because it means that economic progress and scientific advancements will be directly constrained by the quality and volume of available data. As organizations increasingly turn to exclusive datasets, the competitive landscape of AI could change, leading to differentiated models based on unique training corpora. The need for a strategic approach to data collection is paramount; without robust efforts to gather diverse and rich datasets, the advancement of AI capabilities could stagnate despite breakthroughs in algorithms and compute power. The recognition of data as a national strategic asset underscores the importance of proactive data mobilization efforts comparable to large-scale initiatives seen in compute development.