Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D. [pdf] (1password.com)

A recent study by 1Password’s Off-by-1 Labs reveals that Large Language Models (LLMs) like ChatGPT and Claude Code demonstrate significant limitations in generating effective security patches. The researchers tested these models on a series of complex and high-impact vulnerabilities, including notable ones like the "Copy Fail" exploit. Their findings indicate that LLMs achieve a low success rate in fully remediating vulnerabilities without introducing errors or new vulnerabilities, often resulting in superficial fixes that fail to address underlying issues. This raises concerns about the reliability of AI-generated patches in production environments, demanding heightened oversight from skilled engineers. The significance of this research is underscored by the growing reliance on AI assistance in software development, especially for security tasks like patching vulnerabilities. With initiatives such as OpenAI’s Project Daybreak aiming to automate these processes, understanding the efficacy and risks associated with LLM-generated patches is crucial. The study's creation of the FLAWED testing framework serves as a tool for developers to evaluate AI-driven patching outcomes, guiding them away from scenarios where AI is likely to produce faulty patches. This insight is vital for maintaining software security in an increasingly automated coding landscape.