When LLM Decompilers Recompile More and Preserve Less (arxiv.org)

Recent advancements in large language model (LLM) decompilers have significantly improved their ability to generate clean, idiomatic C code from compiled machine code. However, a new study reveals that while these decompilers excel in recompiling code and passing predefined tests, they can overlook critical behavioral discrepancies. Traditional decompilers expose uncertainties as visible placeholders, but LLM-based tools risk failing to preserve crucial functionality—sometimes leading to undetected vulnerabilities. For instance, analysis showed a divergence rate of 4.9% on average, with some systems experiencing up to 13% divergence when tested against a new framework called Decompile-Diverge. To combat these challenges, the researchers introduced a behavioral comparison oracle that generates tests based on function behavior rather than relying on fixed test cases. This approach highlighted that LLM-decompiled code, while achieving a build rate improvement from 75% to 90%, saw a decrease in behavioral agreement from 74% to 62%. The findings underscore the need for enhanced testing methodologies in AI-powered decompilation, emphasizing the risk of overlooking vulnerabilities and functionality losses that can occur when shifting to LLM-based solutions. This work not only challenges the current metrics used for evaluating decompilers but also raises awareness of the critical balance needed between recompilability and preserving original code behavior.