What if automating AI R&D triggers an intelligence explosion? [pdf]

A new working paper from leading AI researchers explores the possibility that automating AI research and development (R&D) could lead to an "intelligence explosion," significantly accelerating AI progress. As AI systems increasingly take over coding and research tasks, current projections suggest that the majority of AI R&D may be fully automated within a few years. This automation could compress years of advancements into mere months, drastically enhancing AI capabilities. However, the authors caution that while this technological leap could bring substantial benefits, it also poses profound risks, including the potential for humanity to lose control over superhuman AI systems.

The paper outlines a feedback loop where improved AI tools not only accelerate the rate of innovation but also expand the R&D workforce by enabling these systems to produce higher-quality successors. Despite potential frictions like diminishing returns and the limits of computational power, preliminary evidence indicates that the momentum toward full automation of AI R&D is gaining traction. The authors urge policymakers to prepare for this rapid change, advocating for measures to steer and constrain the emerging intelligence explosion while ensuring that society can adapt to its implications.