Empirical Computation: Prompting versus Programming [pdf] (mboehme.github.io)

A new vision paper titled "Empirical Computation: Prompting versus Programming," authored by Eric Tang, Jing Liu, and Marcel Böhme, explores the transformative potential of large language models (LLMs) in software engineering. Instead of programming solutions through specific algorithms, the authors propose a method where LLMs solve computational problems via informal prompts. This shift represents a fundamental change in how we approach programming, highlighting the challenges and opportunities that come with this "empirical computation" model. The paper argues against the classical framework of computation and invites the software engineering community to analyze and improve the correctness of LLM-generated solutions. This research is significant for the AI/ML community as it establishes empirical computation as a unique area worth investigating, particularly regarding how LLMs handle various computational tasks like sorting and searching. Preliminary experiments show that while LLMs can sort small arrays with high accuracy, their performance declines as the complexity of the input increases, raising important questions about efficiency and correctness. For instance, the models accurately sort arrays of length 50 over 90% of the time but drop to 58% for arrays of length 150. Understanding these limitations and how to reliably prompt LLMs for computational tasks could redefine the standards of software development in the age of AI.