Literate Programming with LLMs? – A Study on Rosetta Code and CodeNet (research.chalmers.se)

A recent study has explored the potential of Large Language Models (LLMs), like GPT-4 and LLaMA, to facilitate literate programming—a concept developed by Donald Knuth that merges human-readable documentation with machine-readable code. This research, conducted on the Rosetta Code and CodeNet repositories, evaluated LLMs' abilities to generate coherent natural language descriptions alongside corresponding code. By performing computational experiments covering 1,228 tasks across 926 programming languages in Rosetta Code and verifying findings on the broader CodeNet dataset, the study provides insights into how effectively LLMs align programming languages with their semantics. The significance of these findings lies in their implication for software engineering practices. The study reveals that trillion-parameter class LLMs demonstrate a notable capacity for literate programming, whereas models with fewer parameters show a greater proficiency in recognizing programming languages rather than executing tasks. This indicates that the increasing scale and sophistication of LLMs can enhance their utility in software development by encoding the semantics of programming tasks more adeptly. As a result, this research signals important advancements in the integration of AI into coding practices, paving the way for improved software quality through enhanced documentation and code generation.