The Prover, the Skeptic, and the Judge (insertchaos.bearblog.dev)

Recent advancements in vulnerability detection highlight the limitations of current autonomous LLM-powered security tools, which tend to generate false positives along with genuine findings. When tasked with identifying vulnerabilities, these models often provide detailed reports, including proof of concept (PoC) scripts, even when no actual issues exist. This tendency is problematic, as it combines real vulnerabilities with convincingly fabricated ones, leading to potential chaos in security assessments. The challenge arises from the inherent bias of these models to present analyses that affirm findings rather than question them. To address this issue, a new approach has emerged, dividing the roles of vulnerability assessment into three distinct entities: The Prover, The Skeptic, and The Judge. The Prover identifies potential vulnerabilities, while The Skeptic rigorously critiques these claims to determine their validity. Finally, The Judge evaluates the evidence and determines the outcome. By separating these functions, each role can maintain a clear focus and motive—enhancing the accuracy and credibility of findings. This framework not only improves the reliability of security assessments but also supports a structured reporting mechanism, making it easier to differentiate between confirmed vulnerabilities and false positives.