Passing the Test You Trained On: Re-Evaluating Prompt-Injection Detectors (arxiv.org)

Recent research has re-evaluated the effectiveness of prompt-injection detectors used in large language model (LLM) agents, highlighting critical discrepancies between benchmark scores and real-world performance. By testing fifteen detectors, including Meta's Prompt Guard 2, against outputs generated from two agent benchmarks—AgentDojo and tau-bench—the study revealed that the best-performing detector on one benchmark only successfully diagnosed 2% of prompt injections from another benchmark at a low false-positive rate. This indicates significant inconsistencies in detection capabilities that may not be apparent when solely evaluating models on public benchmarks. The findings are crucial for the AI/ML community as they underscore the necessity for more robust evaluation methodologies that reflect actual deployment conditions. Detectors should be assessed using the specific outputs produced by the agents they are designed to protect, rather than relying on generalized benchmark scores. This approach aims to minimize false-positive rates and ensure that detectors are effectively trained on relevant data, ultimately enhancing the reliability of LLM agents in critical applications.