What Happens When Formalization Becomes Cheap? (yangky11.github.io)

A recent exploration from a researcher who began working on machine learning for formal theorem proving in 2018 highlights the evolving landscape of formal verification in AI. With the development of tools like CoqGym and LeanDojo, as well as the capabilities of coding agents such as Codex and Claude Code, formalization has become significantly cheaper. These advancements allow AIs to autonomously formalize complex mathematical claims, such as Fermat’s Last Theorem, and produce software with verified correctness, marking a turning point in making AI outputs more trustworthy. However, this progress has also raised critical questions about the relationship between formal proofs and the actual problems they solve, highlighting gaps between what can be proven and what we need to trust in AI's performance. While formal verification offers an attractive solution to ensuring the reliability of AI systems, particularly in the context of recent AI security concerns, the researcher now cautions against overestimating the assurances these guarantees provide. The experience of the VeriTile project, which aimed to verify GPU kernels for AI training, showcased the challenges of ensuring that optimizations do not change the fundamental behavior of the system. The insights suggest that, despite advancements in automation through AI, many fundamental questions around the specifications and the intent behind AI-generated outputs remain unresolved. As formalization becomes easier, understanding and defining the intent appears to be a critical next step for the AI/ML community in fully leveraging these tools.