Show HN: Spec-shaker – "Chaos engineering" for tests via semantic mutation (github.com)

A new AI-powered mutation testing tool, Spec-shaker, has been announced, designed to enhance the quality of test suites by generating semantically broken code implementations. Unlike traditional mutation testing tools that make random syntactic changes, Spec-shaker utilizes large language models (LLMs) to create realistic bugs that reflect common developer mistakes, such as subtle logic errors and edge case oversights. This innovative approach not only identifies gaps in existing tests but also aims to improve code quality through an iterative process inspired by chaos engineering practices used by companies like Spotify. In a demonstration using an in-memory URL shortener project, Spec-shaker revealed critical shortcomings across three stages of testing. Initially, tests failed to account for important behaviors like handling invalid URLs, leading to a substantial kill rate of surviving mutants. Subsequent iterations showed that, although improvements were made to capture new features, additional gaps remained, such as untested edge cases and logic errors. By the final stage, Spec-shaker successfully identified two survivors out of five new mutants, including a metadata leak, illustrating the continuous need for comprehensive testing even after multiple improvements. Spec-shaker integrates seamlessly into Claude Code, making it versatile across different programming languages and easily accessible for developers aiming to bolster their testing strategies.