Show HN: Evals Coach – a Claude plugin to help PMs write good evals (evalscoach.com)

Evals Coach, a new plugin for Claude, aims to enhance the evaluation process for product managers (PMs) by providing a structured method to create robust evaluation plans. It allows users to convert vague product ideas or requirements into measurable test cases and criteria without requiring a codebase. By importing real outputs, Evals Coach drafts comprehensive evaluation artifacts, including decision thresholds, judge prompts, and test case sets ready for integration with existing evaluation tools. This significantly shifts the evaluation focus from subjective assessments to observable behaviors, ensuring that vital product characteristics are clearly defined and measurable. The significance of Evals Coach lies in its potential to minimize the risks associated with launching AI features that have not undergone thorough evaluations. Often, product decisions are based on vague impressions, leading to recurring issues post-launch. Evals Coach encourages a systematic approach by providing regression cases derived from past failures and helps teams calibrate their grading systems against known benchmarks. This ensures that releases are underpinned by solid evidence rather than assumptions, ultimately improving product quality and user satisfaction. By enabling PMs to articulate specific criteria and behaviors for evaluation, Evals Coach addresses a critical gap in the AI/ML development process, fostering a culture of accountability and precision in product releases.