When AI Writes the World's Software, Who Verifies It? (leodemoura.github.io)

Code Metal has secured $125 million to leverage AI in rewriting software for the defense sector, reflecting a broader trend where tech giants like Google and Microsoft report that 25-30% of their new code is AI-generated. Amazon has successfully modernized large legacy systems using AI, while Microsoft’s CTO predicts that by 2030, 95% of all software code will be produced by AI. Anthropic's creation of a 100,000-line C compiler in just two weeks using parallel AI agents showcases the rapid pace at which AI can produce complex systems. However, this raises significant concerns about the verification of such code, as only a fraction undergo rigorous checks, leading to potential hidden vulnerabilities. The implications of this shift are profound for the AI/ML community, as the balance between speed and reliability becomes critical. Nearly half of AI-generated code fails basic security tests, and without proper verification mechanisms, the risk of systemic vulnerabilities grows, especially when adversaries could exploit AI-generated code. Experts, including Andrej Karpathy, indicate that as humans grow complacent about AI's output, the need for robust specifications and proofs of correctness becomes urgent. To address these risks, the demand for verification mechanisms capable of matching the accelerated generation of code is rising. This could redefine software engineering, transforming verification from an optional process into a necessity for safe and effective AI-driven development across industries.