Show HN: Agent-skills-eval – Test whether Agent Skills improve outputs (github.com)

The launch of **agent-skills-eval** marks a significant advancement in evaluating Agent Skills, an open standard developed by Anthropic to enhance agent performance with domain knowledge. This testing framework allows developers to empirically assess whether their skill enhances model outputs. By running evaluations with and without the skill using a judge model, it generates a comprehensive report that quantifies the skill's effectiveness, addressing a critical gap in the validation process for AI agents. This capability is crucial for the AI/ML community, as it facilitates the systematic improvement of agent performance through data-backed insights. Key technical details include the ability to execute comparisons across multiple prompts and receive structured outputs, including static HTML reports that outline skill pass rates, judge reasoning, and detailed timing metrics. The framework is designed to work seamlessly with OpenAI-compatible models, making it accessible to a broader audience. Additionally, it supports TypeScript SDK and CLI options for easy integration into continuous integration pipelines, enhancing the robustness and transparency of agent skill evaluations. This empowers developers to refine their skills iteratively, leading to more reliable AI applications.