Prompt Snapshot Testing (ninkovic.dev)

In 2025, as large language models (LLMs) become integral to software products, traditional automated testing methods struggle to keep up with their complexity and dynamic nature. At Freeday, the team encountered challenges in testing evolving, multi-part prompts—often personalized and stored across databases—leading to sprawling, time-consuming test suites that slowed down continuous integration pipelines. To tackle this, they introduced a novel approach called "prompt snapshot testing," inspired by visual regression testing in web development. Instead of exhaustively running hundreds of LLM tests, they generate JSON snapshots capturing the prompt’s composition and inputs, comparing these snapshots on subsequent changes to quickly detect meaningful differences. This approach significantly streamlines the development cycle by reducing costs and time—critical in a domain where each token processed has monetary implications. By prompting developers to update snapshots when prompts evolve, Freeday achieves faster iterations and more targeted test execution without compromising thoroughness. While built in-house due to the infancy of LLM testing tools, this method highlights the need for smarter, cost-efficient strategies to validate LLM-driven features as they grow in complexity. The story underscores the ongoing challenge in the AI/ML community of defining effective testing best practices for large, dynamic prompts and invites dialogue on innovative solutions.