Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines (projectzero.google)

In a recent analysis by Project Zero, two automated tools used to identify vulnerabilities in Chrome's JavaScript engine, V8, were compared: Fuzzilli, a generative JavaScript fuzzer, and BigSleep, a bug-finding system powered by a large language model (LLM). Over six months, BigSleep uncovered 45 vulnerabilities, outperforming Fuzzilli’s 37, although there was a 25% overlap in reported bugs. A more detailed experiment suggested a potential overlap of about 60%, illustrating that while both systems are capable of discovering unique vulnerabilities, they often identify different types of issues. This analysis is significant for the AI/ML community as it highlights the evolving role of LLMs in software security, demonstrating their ability to quickly find bugs alongside traditional fuzzing methods. The findings advocate for a complementary use of both approaches, suggesting that vendors should leverage both fuzzing and agentic analysis to enhance their security frameworks. By understanding the strengths and weaknesses of each method, developers can optimize their bug-finding strategies, ultimately improving the robustness of software and safeguarding user data more effectively.