Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code (cacm.acm.org)

Researchers performed a systematic security audit of GitHub Copilot, probing how often the AI pair programmer emits insecure code. They crafted 89 security-relevant scenarios spanning MITRE’s “Top 25” CWEs across software and hardware domains, prompting Copilot to generate 1,689 program completions and analyzing results with GitHub’s CodeQL static analyzer plus manual inspection. Roughly 40% of the generated snippets contained at least one identifiable weakness. The study also shows that vulnerability rates vary with the type of weakness, the exact prompt phrasing (e.g., SQL-injection contexts), and the target domain—hardware/RTL (Verilog) completions were evaluated against hardware CWEs and revealed additional concerns. For the AI/ML community this is notable because Copilot (built on Codex) reproduces patterns from its training corpus of public code, including insecure idioms, and it optimizes for likely completions rather than secure or correct ones. The paper’s three-axis methodology (diversity of weakness, prompt, and domain) and use of CodeQL provide a repeatable framework for evaluating code-model security. Practical implications: developers should treat Copilot’s output as untrusted, apply static analysis and testing, and craft prompts carefully; model builders should prioritize safety-focused fine-tuning, better prompt-conditioning, or integrated security checks to reduce propagated vulnerabilities.