Bootstrapping Fuzzers for Compilers of Low-Resource Language Dialects Using LLMs (arxiv.org)

A new tool called Germinator was introduced, aimed at enhancing the testing infrastructure for extensible compilers of low-resource language dialects. This innovative approach merges dialect-agnostic and dialect-effective methods, allowing for automated test generation that can adapt across various dialects without manual adjustments. Germinator leverages the structural and type constraints encoded in dialect grammars, which are automatically extracted from dialect specifications, and utilizes pre-trained large language models to create diverse seed inputs. The significance of Germinator lies in its ability to improve line coverage in testing by 10-120% compared to existing grammar-based baselines and its success in identifying 88 previously unknown bugs, of which 40 have been confirmed. This marks a substantial advancement in the capabilities of automated testing tools, particularly for low-resource dialects that often lack advanced testing infrastructure. By effectively generating representative test seeds without requiring pre-existing manual training data, Germinator greatly streamlines the process of maintaining compiler correctness, promising improved reliability and robustness in software development for the AI/ML community.