TLAPS-Bench: Formally Proving Correctness with TLA+ Proof System (arxiv.org)

TLAPS-Bench has been introduced as an innovative benchmark and execution harness aimed at facilitating the construction of formal proofs for real-world system specifications using the TLA+ proof system. This tool allows AI agents to tackle the complex tasks of discovering inductive invariants, reasoning about intricate state transitions, and decomposing proofs into manageable components, all while minimizing the potential for reward hacking. By employing TLAPS-Bench, researchers demonstrated its efficacy by proving significant properties in areas such as cache coherence, Byzantine consensus, and write-ahead logs—problems that had previously eluded automated proof efforts. This development is particularly significant for the AI/ML community as it enhances the capability of AI systems to engage in formal proof construction, a critical aspect of verifying complex software systems. The evaluation of leading AI models, including Codex with GPT architectures, on 70 proof targets revealed that the highest-achieving model, GPT-6.1 Sol (max), successfully proved 31 properties, achieving a 44.3% success rate. These findings underscore both the challenges of integrating AI in formal verification processes and the promising possibilities for scaling the use of AI in developing robust, reliable systems.