Can chatbots craft correct code? (blog.trailofbits.com)

At the AI Engineer Code Summit in New York, a prevailing sentiment emerged among AI leaders and engineers: we may soon reach a point where developers rely entirely on chatbots and large language models (LLMs) to write code without ever examining it themselves. This outlook draws a parallel to the transition from low-level programming languages like assembly to high-level languages such as C, but it overlooks a crucial distinction: determinism. While compilers maintain semantic guarantees—preserving the programmer's intent—LLMs function on nondeterminism, yielding varied outputs even with identical prompts. This difference raises significant concerns regarding the correctness and security of the generated code. The implications are profound for the AI/ML community, as the need for determinism is paramount in software development, particularly in critical applications like cryptography where even a slight semantic deviation can lead to vulnerabilities. Recent experiences, such as a developer misdiagnosing a bug due to misunderstandings of context, illustrate the challenges of using LLMs for complex legacy codebases. As LLMs generate code based on probabilistic interpretation rather than concrete rules, they risk making improper modifications, which can complicate rather than simplify the coding process. Moving forward, the community must prioritize integrating formal verification methods and improved frameworks to leverage the benefits of AI while safeguarding software integrity.