Jacob Coxon warns of human extinction and triggers a preference cascade (thezvi.substack.com)

Jacob Coxon, a former researcher at Anthropic, recently resigned in protest, amplifying concerns within the AI community about the race to build superintelligent AI systems. He warned colleagues and the public that current efforts may have catastrophic consequences. In a candid statement, Coxon emphasized that major AI labs like OpenAI and Anthropic are recklessly pursuing self-improving AI without a thorough understanding of the potential risks, including threats to human existence and societal stability. His resignation has triggered a "preference cascade," leading to increased openness among AI researchers about the existential risks associated with their work. Coxon's resignation highlights a growing unease among AI developers, many of whom believe that unchecked progress in AI could result in dire outcomes within a decade. This sentiment is echoed by other industry insiders who are now openly discussing the urgent need for more responsible AI development practices. The unfolding situation reflects a pivotal moment in the AI field, where ethical considerations and safety protocols may require immediate attention, potentially reshaping guidelines for future AI research and development. As awareness grows, this could spur pivotal discussions around the need for international cooperation and regulatory frameworks in AI development to mitigate risks of catastrophic failures.