Show HN: Verdict – model evals on your own data, not someone else's benchmark (github.com)

A new tool called Verdict has been introduced, allowing users to benchmark large language models (LLMs) against their own data instead of relying on standard benchmarks. Verdict enables users to run specific prompts through various models like GPT-5.4-mini, Claude Sonnet, and others, providing a side-by-side comparison of performance based on customizable metrics. This capability empowers users to select the most effective model for their unique tasks and iteratively improve their outcomes through prompt engineering and fine-tuning. The significance of Verdict lies in its democratization of model evaluation, allowing both open-source and closed models to be benchmarked directly against user-specific datasets. It supports various input formats and allows for dynamic model comparisons, including local setups without API fees. Key features include customizable metrics through reference-based or LLM-as-judge evaluations, providing essential feedback for fine-tuning strategies. The tool’s architecture facilitates concurrent processing and error management, making it a versatile addition to the arsenal of AI practitioners looking to optimize their model performance more effectively.