Compiler Testing – Part 1: Coverage-Guided Fuzzing with Grammars and LLMs (nowarp.io)

A recent exploration in compiler testing has highlighted the efficacy of coverage-guided fuzzing using grammars and large language models (LLMs), specifically targeting smart-contract compilers like Sui Move, Cairo, and Solidity. This study uncovered over 100 compiler bugs in structurally valid programs, demonstrating that even mature, audited compilers are not invulnerable to failures originating from valid inputs. This is significant as it showcases the necessity of robust testing frameworks in the evolving landscape of smart contracts, which often involve unique challenges compared to traditional programming languages. The research emphasized the importance of grammar-aware fuzzing to generate syntactically correct inputs that can probe deeper into compiler logic, far beyond merely triggering lexer or parser errors. Utilizing custom mutators informed by tree-sitter grammars and integrated into AFL++, the approach significantly enhances the ability to challenge the compiler's internal invariants. The findings not only underscore the potential pitfalls in compiler design but also advocate for the development of tailored fuzzing tools that adapt to the specific needs of smaller, less popular programming languages, ensuring safer and more reliable smart contract development.