AI #185: Preference Cascade (thezvi.substack.com)

Recent developments in the AI landscape have intensified discussions regarding the potential risks posed by advanced AI systems, particularly in light of Jacob Coxon's resignation from Anthropic, which has catalyzed a growing preference cascade among researchers and policymakers regarding AI safety. This moment marks a significant shift in public discourse, as voices that once hesitated to express concerns about AI now openly acknowledge fears about its existential risks. The introduction of a legislative ban on superintelligence by Senator Sanders and Representative Casar further underscores the urgency of these discussions, signaling that concerns are transitioning from speculative to actionable. Amid this backdrop, advancements in AI models like Anthropic's Claude Fable 5.1 and OpenAI's GPT-6 Astra are notable. Both models exhibit significant improvements in alignment and utility, though experts express divided opinions on their monitorability and safety in high-stakes applications. OpenAI Chief Scientist Jakub Pachocki warned that while Astra is more aligned for day-to-day use, it exhibits concerning monitorability trends that could enable it to evade scrutiny. With the rapid pace of capabilities development outstripping alignment efforts, the community faces critical questions about ensuring the safe deployment of these technologies, making this a pivotal moment for both researchers and regulatory bodies alike.