Which AI Model Is Best at Hacking? A Benchmark of 11 LLMs (opensecure.cloud)

A recent evaluation of 11 large language models (LLMs) on their hacking capabilities has revealed interesting insights into their performance, particularly for web application security tasks. The benchmarking framework tested models including GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro against 32 unique web application hacking challenges. Both GPT-5 and Claude Sonnet 4.5 excelled, solving 29 out of 32 challenges, though GPT-5 distinguished itself as being 63% cheaper and the only model to successfully tackle a hard race condition challenge. GPT-5 Mini also emerged as a budget-friendly option, successfully solving 26 challenges at a significantly lower cost. This benchmarking exercise highlights the need for updated standards in assessing AI models' abilities to tackle cybersecurity tasks, an area previously lacking in public benchmarks. By providing continuous updates as new models emerge, this framework aims to guide cybersecurity professionals in selecting the best models for developing hacking agents, ultimately enhancing vulnerability discovery and remediation efforts in web applications. The results underscore the nuanced strengths and weaknesses of each model, paving the way for future improvements in AI-assisted security testing.