Dear Agent: Prove It (rijnard.com)

Recent advancements in AI have led to the development of techniques that allow large language models (LLMs) to generate formally verified programs. One compelling example is a Lean-written function that counts character occurrences in a string, which can now be automatically proven correct for all possible inputs. This milestone signifies a paradigm shift for the AI/ML community, as it demonstrates that machine-generated code can achieve a level of correctness that previously required extensive testing and verification by human programmers. The key technical implication is that specifications in formal verification can now be generated or guided by LLMs, making the process of ensuring code correctness more accessible. By automating the proof generation, LLMs reduce the manual burden traditionally associated with writing formal specifications. This breakthrough presents an opportunity for future software development, where agents not only write code but also validate its correctness through provable claims, fundamentally changing the landscape of software engineering and paving the way for more reliable and secure software systems. The growing reliability of LLMs in formal verification suggests that, moving forward, computational correctness could become a standard requirement in the development process.