LRTS – Regression testing for LLM prompts (open source, local-first) (github.com)

LRTS (Language Model Regression Testing Suite) has been introduced as an open-source tool designed for regression testing of large language model (LLM) prompts. By comparing outputs from different prompt versions across multiple test cases, LRTS enables developers to assess behavioral changes due to prompt modifications, model upgrades, or parameter adjustments. It provides detailed drift reports, including scores and reasoning, akin to the functionality of testing frameworks like pytest, making it essential for maintaining output consistency and reliability. This tool is significant for the AI/ML community as it directly addresses the challenges associated with prompt engineering and the unpredictable nature of LLMs. With a straightforward command-line interface, LRTS facilitates seamless integration into existing CI/CD pipelines, blocking merges when behavioral drift exceeds predefined thresholds. Moreover, it is designed to be local-first, allowing for rapid testing during development without incurring costs associated with API calls. With minimal dependencies and built-in caching, LRTS not only optimizes testing speed but also enhances test reproducibility and audibility, making it a valuable resource for teams that rely on prompt-based deployments.