Lean 4 software scaling laws (gwern.net)

A new research proposal aims to empirically measure how the perplexity of coding language models (LLMs) scales with the size of codebases, using the Lean programming language as a focal point. This study seeks to determine whether formal languages exhibit better predictability exponents, potentially enhancing the security and reliability of software globally. By analyzing how coding LLMs perform on different programming languages, the researchers hope to uncover insights into which languages might allow for easier code understanding and generation as codebase sizes increase. This work is significant for the AI/ML community as it addresses the pressing issue of software quality in an era where coding LLMs are expected to generate most of our software. Currently, many LLMs produce mediocre-quality or insecure code, raising concerns about their ability to reliably handle large and complex codebases. By measuring and comparing the perplexity across various programming languages, the study could justify investments in rewriting code in languages like Lean, which may lead to more secure software through improved predictability and correctness. The research methodology includes leveraging pretrained LLMs to analyze code quality, detect anomalies, and evaluate the implications of codebase structure on predictability at scale, paving the way for significant advancements in software engineering practices.