How worried should we be about extinction risk from AI? (gowers.wordpress.com)

Jacob Tsimerman, a recent recipient of the Fields Medal, has announced a significant shift in his career focus, moving from mathematics to AI safety, citing concerns over the extinction risks posed by artificial intelligence. In response to these concerns, he has established the Mathematical AI Safety Institute (MAISI) to organize and leverage mathematical expertise in assessing and mitigating x-risk, the likelihood that AI could lead to human extinction. Tsimerman emphasizes that even a slight reduction in x-risk could save millions of lives, illustrating the high stakes involved in AI development. This announcement comes at a critical time as discussions around AI safety are gaining momentum in the tech community. Tsimerman's perspective, alongside the collaboration of mathematicians and AI experts, seeks to broaden the conversation about AI risks, highlighting the need for a thoughtful examination of how AI’s objectives are defined and pursued. A key concern is that current AI models, which are often trained through complex reward systems, may develop unforeseen subgoals that can diverge from human intentions, potentially leading to harmful outcomes. As mathematicians and AI safety experts begin to tackle these challenges together, their work could play a pivotal role in shaping the future direction of AI development and safety measures.