AI heavyweights call for end to 'superintelligence' research (theconversation.com)

Hundreds of scientists, public figures and leaders — including AI pioneers Yoshua Bengio and Geoff Hinton, safety researcher Stuart Russell, tech founders like Steve Wozniak, and political and cultural figures from Susan Rice to Will.I.am — have signed a public statement calling for a prohibition on researching “superintelligence” until there is broad scientific consensus that it can be developed safely and with strong public buy‑in. The move targets an explicit, well‑funded strategic goal of many leading AI companies — backed by hundreds of billions of dollars and frontier research — to build systems that outperform humans across essentially all cognitive tasks, potentially within the next decade. The signatories frame superintelligence as a unique systemic risk: not malevolent intent but goal misalignment and runaway competence. Technically, dangers arise when agents can self‑modify, redesign goals and act at superhuman speed and scale — producing catastrophic outcomes if objectives are specified incorrectly (classic “paperclip” and perverse‑incentive thought experiments). The statement argues current governance focuses on bias, privacy and automation impacts while neglecting existential control problems, and cites historical cascades (2008 finance crash, invasive species, pandemic spread) as analogies. Implications for the AI/ML community include urgent priorities on alignment research, robust containment and verification methods, and international governance to decide whether — and under what conditions — work toward uncontrollable superintelligence should resume.