LLMs Can't Program (functional.computer)

In a thought-provoking exploration, Samir Talwar argues that large language models (LLMs) are fundamentally ill-equipped for programming tasks, stressing that the "generative" aspect of AI often fails to produce meaningful code. His firsthand experiences during a workshop revealed significant shortcomings when using LLMs like OpenAI Codex for software development. Despite generating large specifications and some functional features, the output consistently resulted in flawed, non-compliant code, with a notable incident where an authentication function was improperly implemented. This raises concerns about the reliability of LLMs in programming contexts, where they often generate verbose, superficial outputs instead of genuinely useful code. Talwar emphasizes that while LLMs excel at generating large volumes of text, including code, their value does not lie in full automation of programming tasks. Instead, he suggests that these models may serve as supplementary tools for specific tasks like identifying discrepancies or edge cases in code, akin to enhanced linters. He points out that LLMs can complement human efforts in code review by offering instant feedback, thereby addressing the shortcomings of human programmers. Ultimately, Talwar advocates for a more realistic understanding of LLM capabilities, suggesting they should be viewed as mechanical aids rather than replacements for human expertise, particularly given the increasing complexity of programming challenges.