Evaluating Agentic Cybersecurity in Attack/Defense CTFs: Offensive Is Not Better (arxiv.org)

Researchers tested whether autonomous AI agents are inherently better at attacking or defending in cybersecurity by deploying agentic systems via the CAI (Cybersecurity AI) parallel execution framework across 23 Attack/Defense CTF battlegrounds. In unconstrained scenarios, defensive agents achieved 54.3% success at patching versus 28.3% for offensive initial access (p = 0.0193). However, when realistic operational constraints were imposed—defenders required to maintain availability (success 23.9%) or to prevent all intrusions (15.2%)—the defensive advantage disappeared (no significant difference, p > 0.05). An exploratory taxonomy of exploited vulnerabilities hinted at recurring exploitation patterns, though limited sample sizes limit firm conclusions. The study’s controlled, empirical comparison challenges prevalent claims that AI gives attackers an inherent edge and highlights that success metrics and operational constraints fundamentally shape outcomes. For the AI/ML and security communities this means evaluation frameworks must mirror real-world goals (availability, completeness of defense) rather than simple binary win/loss measures. Practically, the results underscore urgency for defenders to adopt open-source Cybersecurity AI tools and rigorous, constraint-aware benchmarks to keep pace with offensive automation, and they call for larger-scale, standardized CTF evaluations to clarify which agentic strategies scale reliably in realistic deployments.