Can open-source prompt-injection detectors catch realistic AI agent attacks? (github.com)

A recent study tested ten open-source prompt-injection detectors against 629 real-world AgentDojo injection attacks embedded within typical tool outputs, revealing significant challenges in effectively detecting these sophisticated attacks. The best-performing detector, jailbreak-detector-large, managed to detect only 51% of the attacks but generated a concerning 2% false positive rate by blocking benign traffic. In contrast, the well-known Meta's Prompt Guard 2 barely caught 1% of attacks, emphasizing the inadequacies of current models in distinguishing harmful instructions from harmless ones. This evaluation is crucial for the AI/ML community as it highlights the limitations of existing detection tools in realistic environments, particularly where malicious commands mimic regular user requests. The findings suggest that the true danger lies in the subtlety of agent attacks, which are often designed to blend seamlessly into normal interactions. The researchers recommend that defense mechanisms should implement policy-based controls, prioritizing context-aware enforcement rather than solely relying on text-level detection, as current models are prone to dilution from benign surrounding data. This insight could steer future developments in safeguarding AI agents against increasingly nuanced injection attempts.