CVE-Bench: testing LLM agents on real-world vulnerability patches (giovannigatti.github.io)

CVEBench has been introduced as a new benchmark for evaluating large language model (LLM) agents on their ability to address real-world security vulnerabilities, building on previous claims by Anthropic that its models can outperform human experts in finding such vulnerabilities. This initiative tests five different models against twenty real-world Common Vulnerabilities and Exposures (CVEs) across three distinct task conditions: using advisories, diagnosing issues without precise locations, and locating issues without descriptive contexts. Each model is assessed in a controlled environment where it must manipulate code to repair vulnerabilities, with a focus on genuine reasoning abilities rather than simple instruction-following. The significance of CVE-Bench lies in its rigorous approach to evaluating the capabilities of AI models in a crucial area of software security. By emphasizing real-world security issues and stripping away shortcuts like access to fixes, CVE-Bench illuminates whether AI models can independently understand and rectify vulnerabilities. The results show that no model consistently fixes real vulnerabilities, with the highest scorer, GPT-5.5, managing to solve 50% of tasks overall—underscoring the challenges that remain in leveraging AI for effective security solutions in software development. This benchmarking effort promises to guide future model improvements and enhance security in the AI/ML community.