AI Red Teaming Guide (github.com)

A new, practical AI Red Teaming Guide offers a comprehensive playbook for adversarial testing and security evaluation of AI systems, aiming to surface vulnerabilities before attackers do. Grounded in real-world experience (Microsoft’s 100+ product red teams) and aligned with NIST AI RMF, OWASP GenAI, MITRE ATLAS and CSA guidance, the guide is targeted at security teams, ML engineers, risk/compliance officers and organizations deploying high‑risk AI. It stresses why red teaming matters now: LLMs and agentic systems are moving into healthcare, finance and infrastructure, recent incidents (2024–2025 prompt injections, data leaks and ChatGPT exploits) highlight tangible harms, and regulations such as EU AI Act Article 15 and the U.S. Executive Order require demonstrable robustness and cybersecurity testing. Technically, the guide lays out an operational methodology—scope definition, MITRE ATLAS threat modeling, risk‑profiling, and test-plan development—covering access modes (black/gray/white box), testing approaches (manual, automated, hybrid) and measurable metrics for fairness, robustness and trustworthiness. It catalogs attack vectors (jailbreaking, prompt injection, model extraction, data poisoning, adversarial examples) and agentic risks (permission escalation, memory manipulation, orchestration flaws), recommends toolkits (NIST’s Dioptra testbed, OWASP blueprints) and advocates purple‑team collaboration, continuous monitoring, and incident response. The guide emphasizes practical, iterative red teaming—human creativity plus automation—to find emergent, context‑dependent failure modes that traditional security testing misses.