Show HN: Crucible – your AI wrote the tests, so who tested the tests? (github.com)

Crucible, a new tool introduced on HN, aims to enhance testing practices in AI-generated code by addressing the critical question: if your AI wrote the tests, who is testing those tests? This innovative framework employs mutation testing to evaluate how effectively test suites catch real bugs. With an impressive 97% line coverage, Crucible’s analysis revealed that 25 out of 71 injected defects survived initial tests, underscoring the necessity of reassessing traditional coverage metrics. Notably, Crucible not only identifies these gaps but also generates additional tests to specifically target the missed defects, demonstrating a comprehensive feedback loop in testing. The significance of Crucible lies in its ability to provide developers with actionable insights into the shortcomings of their existing test suites without relying on models or API keys, thus ensuring transparency and cost-effectiveness. The tool operates entirely within the repository, allowing users to compute mutation scores and identify survivors—essentially bugs that went undetected by current tests. This capability not only promotes better testing standards but also addresses the challenge of model nondeterminism, as it retains detailed records of testing outcomes. By integrating these methodologies, Crucible is set to reshape how AI testing is approached, making it a valuable resource for the AI and ML community.