Show HN: TheoremSmith: Mass-Producing Hard Proof Problems to Train LLMs (charleshuang.me)

TheoremSmith has introduced a groundbreaking pipeline designed to automate the generation of complex formal verification tasks aimed at training large language models (LLMs) for improved formal verification capabilities. By leveraging synthetic environments, TheoremSmith can generate difficult proof problems that take several hours to solve, a feat that has historically been constrained by the high costs of formal verification. This advancement is significant for the AI/ML community as it possesses the potential to enhance the accuracy and reliability of both mathematical proofs and software code, thus addressing the increasing incidence of software bugs and outages. The system works by creating seed tasks from existing Lean projects, utilizing chains of dependencies in mathematical proofs to ensure models tackle substantial challenges. A key technical innovation is the integration of Lean Comparator, which safeguards against reward hacking by validating AI-generated proofs against the original problem statements. The success of TheoremSmith in producing tasks that fulfill rigorous quality checks indicates a promising future for automating formal verification, which could transform training techniques and lead to breakthroughs in both mathematics and software engineering. As the landscape of AI-driven formal verification evolves, future enhancements are expected to expand its applicability across various verification tools and real-world coding scenarios.