The Third Risk (thirdrisk.org)

The recent discourse on artificial general intelligence (AGI) has largely focused on two prominent risks: a financial crash resulting from AGI failing to deliver or a loss of human control if it succeeds. However, a more pressing concern, referred to as the "third risk," has emerged: the possibility of an AGI that is perfectly aligned and controlled by a select group of private CEOs. This scenario implies that this small elite could monopolize AGI's abilities, effectively creating a form of "AI feudalism." As a result, they could operate highly advanced technologies and solve complex problems, all while disenfranchising the broader population, who would lose both economic value and autonomy. This risk is significant for the AI/ML community as it highlights the urgent need for transparent discussions about power dynamics surrounding AGI. Unlike past historical rulers who needed the labor of their subjects, an elite controlling AGI could thrive without reliance on the general populace, potentially leading to societal neglect or widespread economic despair. The implications of this scenario necessitate proactive measures, including the establishment of antitrust regulations and open governance to ensure accountability and mitigate the concentration of power in the hands of a few. If these critical conversations are not initiated now, the future may risk aligning too closely with dystopian predictions where humans remain beholden to the whims of powerful AGI operators.