Superintelligence – Why the people building it keep warning us about it (byjorin.substack.com)

The recent discussions surrounding Nick Bostrom's book "Superintelligence: Paths, Dangers, Strategies" have highlighted significant concerns about the rapid evolution of AI towards Superintelligence (SI) and its implications for humanity. Bostrom, who penned the book back in 2014, articulated notions of fast take-off scenarios and alignment issues that are now paramount as researchers race towards creating Artificial General Intelligence (AGI). The focus on SI raises alarm bells about potential existential risks, as advancements in AI technology could dwarf human intelligence by a factor of 1000 or more, questioning our ability to maintain control. This paradox is particularly striking: those promoting AI innovation simultaneously warn of its catastrophic potential, suggesting that SI's arrival is inevitable, and urging caution and regulation to mitigate these risks. The notion that these creators can justify their increasing power by framing it as a protective measure against SI emphasizes the need for conscious choice in AI development. Experts argue for the urgency in implementing guardrails and regulatory measures, advocating that humanity's future depends on responsible action rather than a blind race towards unregulated progress. This collective narrative must shift towards fostering real alternatives and encouraging informed decisions about the trajectory of AI technology.