Evaluating LLM-Generated Detection Rules in Cybersecurity (sublime.security)

Sublime published a tailored evaluation framework and results for LLM‑generated cybersecurity detection rules, answering the practical question: can an AI agent actually produce rules that reduce risk without adding noisy alerts? The framework measures every generated rule across three pillars — Detection Accuracy (precision, unique true positives, and net‑new coverage as a practical proxy for recall), Robustness (an AST‑based score that favors behavioral logic over brittle IOCs), and Economic Cost (pass@k paired with cost‑to‑pass, time‑to‑production, and runtime cost/1k messages). They test rules written in their expressive MQL DSL (which can invoke NLU, computer vision, historical behavior, and ML risk scoring) and compare ADÉ, their Autonomous Detection Engineer, to human engineers. Key findings: ADÉ tends to produce very high‑precision, surgical rules that close detection gaps quickly and surface new signals for behavioral ML, while human rules often have broader recall. Robustness scores for ADÉ rules were comparable to human rules, and pass@k economics were favourable (median cost‑to‑pass ≈ $1.50 at k=1, ≈ $4.30 at k=3), enabling predictable spend and fast time‑to‑production. Limitations include current reliance on static analysis rather than adversarial dynamic testing—plans to add adversarial inputs and expand to spam/graymail are underway. The work reframes evals from vague benchmarks to operational metrics, providing a repeatable, business‑oriented way to trust and iterate on AI detection agents (full paper on ArXiv; CAMLIS presentation planned).