Show HN: Caliper – pass@k reliability testing for Claude Code and Codex skills (github.com)

Caliper, a new reliability testing tool, has been launched for evaluating the skills of AI models like Claude Code and Codex. This tool enables developers to run their skills multiple times (k times), obtaining a pass@k score that assesses how well a skill performs compared to the base agent. For example, it can measure attributes like the accuracy of generating commit messages or valid configuration files, letting developers track improvements or maintain reliability over time. By establishing a baseline for performance, Caliper can clearly demonstrate the benefits (or lack thereof) of integrated skills. This announcement is significant for the AI/ML community as it addresses a common challenge: the unpredictable nature of model behavior following updates or changes in prompts. Caliper provides a structured method to define success criteria and evaluate skills systematically, giving developers valuable insights into their effectiveness. With technical features like an interactive spec generation process and compatibility with several agents (Claude Code, Codex, and Pi), Caliper is positioned to enhance the reliability and transparency of AI skill development. Overall, it promises to streamline evaluation, making it easier for developers to iterate and improve their AI-driven applications.