Booz Allen Cyber Weapon Index (www.boozallen.com)

The Booz Allen Cyber Weapon Index (CWI) introduces a groundbreaking benchmark for assessing the real offensive capabilities of AI systems in live cyber environments. This initiative evaluated 18 leading large language models (LLMs) from the U.S. and China by deploying them as autonomous attackers against a production-grade enterprise network. Unlike traditional evaluations that focus on theoretical performance, the CWI tested these models under identical, uncurated conditions, allowing them to operate without pre-defined tools and scaffolding. This approach enabled a robust assessment of each model's actual behaviors in real-time, validating their actions through comprehensive network telemetry and logs. This assessment is significant for the AI and machine learning community as it shifts the conversation from theoretical capabilities to real-world applications in cybersecurity. By capturing how well these models can adapt, progress through intrusions, and generate new offensive capabilities, the CWI provides crucial insights into the effectiveness and readiness of AI systems in cybersecurity roles. The findings from this benchmark could guide the development of more resilient defenses against AI-driven cyber threats, ultimately enhancing security measures in enterprise environments.