Gradual Disempowerment: Systemic Existential Risks of Incremental AI Development (arxiv.org)

A recent paper titled "Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development" explores the potential dangers posed by incremental improvements in AI, advancing the notion of 'gradual disempowerment'. Unlike the sudden takeover scenarios often highlighted in AI safety discussions, this research examines how even small advancements in AI can erode human control over essential societal systems, including the economy, culture, and governance. The authors argue that as AI increasingly assumes roles previously held by humans, it may weaken both formal mechanisms of control, such as voting, and the informal societal alignments that depend on human participation, ultimately leading to outcomes misaligned with human interests. This paper is particularly significant for the AI/ML community as it urges a rethinking of how incremental AI developments may jointly reinforce systemic risks across diverse domains. The authors advocate for the necessity of focused technical research and governance strategies that account for the subtle yet profound impacts of AI on human agency. By highlighting the interconnectedness of societal systems, the study emphasizes the precarious balance between technological advancement and maintaining human influence, suggesting that a failure to address these risks could lead to humanity's irreversible disempowerment and potential existential threats.