P(doom) (lucumr.pocoo.org)

This week, a viral discussion around the possibility of AI leading to catastrophic outcomes reignited concerns within the AI and machine learning community. Notably, Dario Amodei assigned a probability of 10-25% to severe risks posed by current AI systems, prompting reflections on AI safety and regulatory failures in handling these technologies. Amodei's warnings about persistent botnets and the blind operation of major labs like OpenAI and Anthropic paint a troubling picture, considering the potential for misuse and the existing strain on public resources caused by proprietary models. The significance of this discourse lies in its implications for AI governance, especially as it suggests a singular focus on a few dominant players, potentially sidelining broader community interests. The concern is that innovation and accountability are hampered by a concentration of power in a handful of corporations, which risk creating a "token economy" reminiscent of an underground market. Furthermore, calls for third-party evaluation of AI models highlight the need for transparent practices, as current closed models raise ethical questions and exacerbate inequalities in access and benefits. As AI capabilities grow, the fear is not only about existential threats but also about the deepening challenges faced by researchers and developers outside elite labs.