AI-powered security engineers and source code scanners (joshua.hu)

An independent follow-up test of modern “AI security engineer” source-code scanners shows these tools are maturing but uneven. The author re-ran and expanded prior tests across several products: ZeroPath stood out, finding ~150–200 real issues in curl (and driving many fixes), while DryRun scored near the top for PR scans. Several open-source projects—Fraim (hobby, hard to set up), LAST (POC that funnels files to a commercial LLM), and metis (Arm’s RAG-based tool that chunks/indexes code into a DB like ChromaDB, retrieves context, prompts language-specific templates, supports C/C++/Python/Rust)—were reviewed. Ghost Security (web-app focus) and others showed high false-positive rates, flaky tooling, and immature integrations; Endor Labs focuses on AI-generated-code vulnerabilities and dependency-CVE triage. HackerOne Code and TerraSecurity represent PR/DAST-style approaches with varying transparency. Technically this confirms two things: LLM-enabled static analysis can surface real, non-trivial bugs at scale (especially when paired with good corpus/indexing and prompt engineering), but results depend heavily on pipeline design—embeddings + RAG, language-specific prompts, and FP suppression matter. Many tools still struggle with hallucinations, noisy outputs, or limited scopes (PR-only vs full-repo, static vs dynamic). Practical adoption therefore favors a hybrid workflow: automated AI scans to surface candidates, careful FP reduction, and human triage for context-sensitive or logic/design flaws. The landscape is promising, but product maturity and precision remain the gating factors.