Grok 4.6 and GPT5.6 beat Anthropic for finding security vulnerabilities in PRs (docs.damsecure.ai)

In a recent benchmarking study, OpenAI's GPT-5.6 emerged as the top model for identifying security vulnerabilities in pull requests (PRs), outperforming Anthropic's offerings and other competitors like Fable 5 and Claude Sonnet 4.6. The study tested ten models on ten PRs, each containing a specifically planted access-control bug. GPT-5.6 achieved a remarkable 100% recall rate, ensuring all vulnerabilities were detected, while also maintaining cost-effectiveness at just $0.70 per PR. In stark contrast, Fable 5 was found to be significantly less efficient, landing below the cost/quality frontier with a cost per PR of approximately $3.61 and substantially lower performance metrics. This finding is significant for the AI/ML community as it underscores the importance of model selection in specific workloads like PR security reviews, which are prevalent among software developers. The researchers employed rigorous testing methods using synthetic PRs to eliminate biases and ensure accurate results, highlighting both precision and cost-effectiveness as vital factors in model performance. With GPT-5.6 dominating in recall and overall efficiency, it sets a new standard for AI tools used in security reviews, emphasizing a need for developers to be discerning about their choice of AI tools to avoid unnecessary costs and inefficiencies.