Show HN: AgentToolBench-Code – security benchmark for AI coding agents (gist.github.com)

A new open-source benchmark, AgentToolBench-Code v0.0.1, has been introduced to evaluate the security of AI coding agents, specifically focusing on silent security failures. The benchmark has been expanded from 10 to 16 scenarios, revealing significant performance differences between two AI models, Claude Code Sonnet 4.6 and Haiku 4.5. While both models initially tied on the original corpus, the expanded set highlighted Sonnet's superior capability to recognize and address potential security threats, such as detecting PyPI typosquats and internal IPs, outperforming Haiku by six score points (Sonnet: +9, Haiku: +3). The technical implications of this benchmark are extensive, as it offers a robust framework for assessing model vulnerabilities against real-world attack scenarios, emphasizing the importance of capability scaling in enhancing AI's threat recognition abilities. The new benchmark not only serves as a critical tool for developers to evaluate AI security but also fosters an ongoing dialogue about classification accuracy and security weaknesses within AI systems. This initiative promotes community engagement, calling for contributions to enhance the benchmark's scenarios and accuracy, ultimately aiming to support the safety and reliability of AI applications.