The Engineer and the Autocomplete: What LLMs Can't Do (geerttheys.substack.com)

The article explores the differences between human software engineers and large language models (LLMs) in their approach to coding and problem-solving. While LLMs excel in speed and surface area—having encountered vast amounts of code and common patterns—they fundamentally lack the causal ownership that human engineers possess. This ownership allows engineers to interrogate causal relationships, hypothesize about failures, and maintain a deep understanding of their systems, something LLMs cannot replicate due to their ephemeral in-context memory. The implications are significant for the AI/ML community, particularly as it increasingly integrates LLMs into software development. While LLMs can efficiently handle routine tasks and provide rapid solutions, engineers must remain vigilant in maintaining their mental models and critical thinking. When paired correctly, human ingenuity can leverage LLMs for mundane tasks, but there is a risk that reliance on LLM-generated solutions may lead to a superficial understanding of complex systems. This dynamic underscores the importance of combining human creativity with LLM capabilities to enhance productivity without sacrificing fundamental comprehension.