When Modernization Breaks Working Code: A Study of LLMs on Legacy PHP (arxiv.org)

A recent study titled “When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP” highlights significant challenges with using large language models (LLMs) to modernize outdated PHP code, specifically targeting the transition from WordPress 4.0 to PHP 8.3. Researchers tested five LLMs on 100 legacy PHP files, evaluating their outputs against 630 hidden behavioral checks and modernization benchmarks from the tool Rector. The findings revealed that while LLMs attempted to update the code, they often compromised functioning files, breaking between 32% to 51% of them, while successfully modernizing at most 10. In stark contrast, the Rector tool broke only 7 files while repairing 6, underscoring the risks associated with relying on LLMs for code modernization without project context or execution feedback. This study is significant for the AI/ML community as it exposes the limitations of LLMs in handling legacy systems, suggesting that their outputs can introduce errors rather than facilitate improvements. The results call for an enhanced evaluation approach for LLM-based modernization efforts, advocating for comparisons against a no-change baseline while separately measuring modernization efficacy and behavioral preservation. The implication is clear: without proper context and guidelines, LLMs may not only fail to improve legacy code but may also introduce unnecessary complications, highlighting the need for cautious application in software engineering practices.