CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat (arxiv.org)

CyberArena, a newly introduced benchmark for AI agents, addresses a critical gap in evaluating cybersecurity capabilities by simulating dynamic, real-time attack-and-defense scenarios. Unlike traditional benchmarks that assess agents in isolation against static targets, CyberArena facilitates multi-round contests where agents alternate between offensive and defensive roles within shifting environments. This dynamic setup allows for a more realistic assessment of an agent's ability to adapt under pressure, exploiting vulnerabilities while simultaneously defending against them. The significance of CyberArena lies in its ability to more accurately measure the performance of AI agents in cybersecurity, moving beyond the limitations of single-agent baselines that often plateau. Through the evaluation of 12 leading large language model (LLM) agents across 40 challenges, the system generates multidimensional scores reflecting offensive, defensive, and domain-specific abilities. This innovative approach not only enhances the benchmark's relevance to real-world cybersecurity scenarios but also creates a scalable platform for future research, helping to develop more effective and capable cybersecurity agents that can think strategically and act autonomously in live combat situations.