Can a local LLM beat -O3? (tijl.blog)

During the recent ACCU conference, Andrei Alexandrescu highlighted a groundbreaking concept: the integration of local Large Language Models (LLMs) with compilers to optimize code. This intriguing prospect explores whether small, on-premise LLMs can outperform traditional compiler optimization flags like -O3. At CppCon, a poster presentation shared experimental results from a unique project aimed at evaluating the performance gains and functional correctness of LLM-generated optimizations within the compilation process, specifically using the Clang/LLVM framework. Tests included various C++ functions designed with intentional inefficiencies to challenge both the compiler and LLMs. The findings indicate that while conventional compilers often yield impressive optimizations, LLMs can propose alternative solutions that may produce faster code, especially when addressing deliberate inefficiencies in the test programs. However, many LLM-generated codes were not functionally correct or failed to compile. The experiments also revealed significant challenges such as response truncation and context limitations affecting LLM performance. Overall, this exploration into LLM-assisted compilation not only showcases exciting possibilities for optimizing code but also raises questions about the balance between local and cloud-based ML solutions in enhancing software development.