The Safety That Isn't (markatwood.substack.com)

A recent experience with AI tools has highlighted significant flaws in the safety systems that govern their use in security work. While performing FIPS 140-3 compliance checks, an engineer utilized Claude agents to identify real vulnerabilities in production software, including critical issues like authentication bypasses. However, when the engineer sought assistance in filing a responsible security disclosure with GitHub, the AI's safety mechanisms blocked this request, categorizing it as potentially dangerous despite the benign intent. This scenario underscores a crucial failure: the AI's inability to differentiate defensive activities, such as vulnerability reporting, from offensive ones. This incident raises alarming questions about the policies surrounding AI safety, which appear heavily influenced by fear-driven narratives. Two groups—the so-called doomers focused on existential AI risks and intelligence community alumni wary of dual-use technologies—are crafting these policies without direct experience in cybersecurity practices. Their initiatives may actually compromise security by preventing responsible disclosures, thereby protecting their own narratives rather than enhancing safety for users. The result is a troubling dynamic where automated systems designed to safeguard against misuse inadvertently hinder professionals working to secure systems, ultimately affecting the overall security landscape.