Bootstrapping Mechanical Correctness (artagnon.com)

Anthropic recently announced the successful formalization of Fermat's Last Theorem through a collaborative effort involving a team of agents, resulting in over ten million lines of Lean code. This significant advancement raises critical questions about the verification of such extensive automated proofs, as human oversight remains essential to confirm the absence of errors or "unsoundness bugs." The challenge of establishing a completely trusted basis for automatic proof verification highlights the complexities involved in ensuring the reliability of mathematical and computational artifacts. The implications for the AI and ML communities are profound, as this endeavor exposes the limitations of current proof assistants, which require human expertise for validation much like informal mathematical proofs. The need for ongoing verification of both software and hardware stacks—as they receive updates—creates a scenario where full verification is increasingly difficult, if not impossible. The discussion points to an intriguing hypothesis: could a full mechanical verification system eventually necessitate replicating human reasoning in silicon? This underscores the intricate relationship between trust, verification, and the evolving capabilities of AI in formalizing complex mathematical truths.