CodeCrucible: A blueprint for LLM-driven SAST (engineering.block.xyz)

Today marks the launch of CodeCrucible, an innovative LLM-driven static application security testing (SAST) tool designed for uncovering vulnerabilities in codebases. While the tool itself is significant, its broader impact lies in the accompanying blueprint that outlines the crucial design decisions, tradeoffs, and reference implementations. This blueprint empowers other teams to tailor and enhance their own integration processes, addressing common pitfalls in static analysis integration, such as adaptation to varied environments and architectural constraints. By focusing on reusability, the CodeCrucible project aims to foster iterative improvements across the SAST landscape. At the technical heart of CodeCrucible is a whole-repo analysis architecture that diverges from traditional SAST systems, allowing the LLM to conduct primary analysis rather than acting only as a validator for prior findings. The tool efficiently handles large repositories by utilizing a combination of compaction strategies, multi-stage identification processes, and a structured output format that accommodates the limitations of token budgets. Key innovations include leveraging whole-repo concatenation for context-rich analysis, advanced filtering to prioritize relevant code, and separate, phased prompts to ensure optimal token usage. These strategies challenge previous assumptions about LLM performance and present a more effective model for vulnerability detection, enhancing both efficiency and accuracy within modern development workflows.