Nondeterminism's not the problem (isaacvando.com)

A recent discussion challenges the common misconception that nondeterminism is the primary issue affecting large language models (LLMs) when generating code. While LLMs are often described as comparable to compilers, skeptics argue that LLMs' inherent nondeterminism—where outputs can vary even with the same input—makes them less reliable. However, the author contends that the crux of the problem lies not in nondeterminism but in the lack of formal semantics governing prompts, unlike programming languages, which have well-defined rules and specifications. With programming languages, there are guarantees about behavior and output, while with prompts, there are none, leading to unpredictable results regardless of whether the LLM is deterministic or nondeterministic. The piece argues for a nuanced understanding of both LLMs and compilers and suggests that even deterministic LLMs can produce unreliable outputs because they lack the structured semantic constraints found in traditional programming paradigms. It highlights that merely tweaking LLMs for deterministic behavior does not resolve the fundamental issues of trust and reliability in the artifacts generated from vague prompts. The author proposes that more robust tools may be necessary to validate LLM outputs against expected semantics, although this could complicate the user-friendly nature that makes LLMs appealing.