Finding Bugs Using LLMs (materialize.com)

Materialize has successfully leveraged Large Language Models (LLMs), specifically Anthropic’s Opus 4.7, to identify bugs in code and open pull requests since early 2026. Their system employs a basic shell script to direct a coding agent to analyze various code units, including new pull requests ready for review, historical commits, and production source files, thus catching bugs before code merge and backfilling their repository's history. By categorizing bugs by severity and focusing only on high and medium risks while avoiding known issues, Materialize has been able to streamline the bug detection process. This approach has proven effective, uncovering hundreds of valuable bugs that past test suites overlooked. This development is significant for the AI/ML community as it showcases LLMs’ potential in software quality assurance, pushing the boundaries of traditional testing methods. While Opus 4.7 offered fewer false positives due to its contextual analysis, the challenge remains in handling API restrictions imposed by providers when LLMs are used for vulnerability assessments. Moreover, the project underscores the importance of human verification, as developers still need to meticulously validate findings to avoid misclassification and ensure effective bug resolution. Ultimately, while LLMs enhance bug-finding capabilities, they do not replace systematic testing, revealing gaps that could lead to improved testing frameworks.