Big Blob of Compute: The Essay That Started the AI Race (kevinroose.substack.com)

In 2017, OpenAI researcher Dario Amodei introduced the transformative idea that scaling AI models—making them larger and training them with more data and computational power—could significantly enhance their intelligence. This hypothesis, captured in his unpublished essay “The Big Blob of Compute,” suggested that improvement in AI capabilities might not require intricate algorithms but simply more raw computational resources and a conducive environment for training. Amodei's insights aligned with observations from biology, where larger-brained animals generally exhibit higher intelligence. This radical shift in thinking encouraged the AI community to pivot from developing narrow models to scaling up general-purpose architectures, ultimately leading to the creation of widely influential models like GPT-2 and GPT-3. The significance of Amodei's hypothesis cannot be overstated; it essentially ignited the current AI race by fostering a focus on scaling that has shaped the trajectory of machine learning advancements. His document has since gained a mythical status within the AI community, influencing the perspectives of many researchers who came to prioritize computational scale over complexity in model design. By allowing AI models to learn and adapt within rich training environments rather than micromanaging their training processes, Amodei’s ideas have helped lay the groundwork for the capabilities displayed in today’s AI systems, highlighting a critical paradigm shift in AI development and safety strategies.