Finding Miscompiles for Fun, Not Profit (newsletter.semianalysis.com)

A recent project by a compiler engineer revealed the potential of AI in bug-finding within compilers. Over a few days, they used a combination of fuzzing and large language models (LLMs) to identify numerous bugs in NVIDIA's low-level compiler, ptxas, and LLVM’s AMDGPU backend. Their approach significantly accelerated the bug discovery process, highlighting how AI can enhance software testing and security efforts. The findings are significant for the AI/ML community as they illustrate a new paradigm in automated software debugging—using AI to automate the tedious aspects of code analysis and bug detection. By employing LLMs to create sophisticated fuzzers and subagents, the engineer discovered about 80 miscompiles in ptxas and continued to find bugs at an impressive rate of one every four minutes in LLVM’s x86 backend. Despite the high costs associated with this method, the discovery of severe bugs, including instances of data corruption, underscores the necessity of leveraging AI tools in ensuring the integrity of advanced software systems. This level of automation represents a major leap forward, suggesting that, with the right resources, automating code inspection may soon surpass traditional debugging methods.