Why AI researchers keep building something they think will kill humans (www.businessinsider.com)

At a recent Goldman Sachs tech conference, OpenAI CFO Sarah Friar revealed that the company's most advanced AI models can now train smaller models, marking a significant leap in recursive self-improvement (RSI). This capability, while seen as a huge business opportunity by investors, has sparked fear among AI researchers who worry that rapidly advancing AI systems could become dangerously powerful. The notion that AI could potentially pose existential threats is echoed by figures like Evan Hubinger from Anthropic, who alarming predicts a substantial risk of AI causing human extinction within the next decade. This tension highlights a complex dilemma within the AI community. Researchers at companies like Anthropic and OpenAI express a sense of moral responsibility to ensure safe AI development, fearing that if they pull back, less responsible entities might take the lead. This internal struggle, compounded by feelings of lost agency in the face of rapid AI advancements, has led some to voice ominous predictions publicly. The ongoing dialogue questions why AI developers persist in their work amid such contrasting fears, pointing to commercial incentives and a belief in their unique capability to manage potential risks as driving factors. As AI technology evolves, the balance between innovation, safety, and ethical responsibility becomes increasingly precarious, reflecting deep anxieties within the industry.