Show HN: An open source benchmark for prompt-injection detectors (github.com)

Bastion Soft has announced the launch of a model-agnostic open-source benchmark designed for evaluating prompt-injection detectors. Unlike traditional benchmarks that only measure a detector's ability to flag attacks, this new framework assesses both the attack catch-rate and the false-positive rate on real traffic, ensuring a more comprehensive evaluation. This threshold-agnostic approach allows for a more reliable comparison, as the detectors are ranked based on consistent metrics at a fixed detection rate instead of arbitrary thresholds. For the AI/ML community, this benchmark represents a significant advancement in assessing the robustness of prompt-injection detection systems, which have become increasingly vital in the era of large language models (LLMs). Bastion’s methodology is fully reproducible without the need for a GPU, making the data accessible to researchers and developers alike. By allowing contributions from other projects and emphasizing the open nature of its results, this benchmark fosters collaboration and honesty in evaluating detector performance, ultimately improving the security measures against prompt-injection attacks.