CompletionKit – stop guessing whether your prompt change helped (completionkit.com)

CompletionKit is a new open-source tool designed to enhance the evaluation and performance of AI applications, allowing developers to test prompt changes with confidence. It provides a systematic approach to improving prompts through real input testing, scoring outputs against customizable metrics like empathy and clarity, and facilitating iterative improvements. By eliminating guesswork—"shipping on vibes"—CompletionKit enables developers to base decisions on rigorous evidence rather than intuition, addressing the critical issue of prompt drift and regression in production environments. This tool is significant for the AI/ML community as it centralizes the evaluation process, supporting multiple AI models—including those from OpenAI and Anthropic—through an easy-to-use interface. Users can run their tests in three different deployment options: hosted cloud, standalone app for self-hosting, or integrated as a Rails engine within existing applications. With its focus on transparency and reproducibility—providing versioned prompts and allowing custom scoring—CompletionKit aims to set new standards for prompt evaluation, ensuring that AI applications perform reliably as they evolve.