Show HN: jevals – replacing LLM judges with typed Jev decisions (github.com)

A new tool, jevals, has been introduced to enhance the evaluation of AI agent performance by replacing traditional LLM judges with Jev-style decision models. This approach allows scores for various evaluation metrics to be processed in one request, reducing costs drastically—only a few thousandths of a cent per trace—and improving response time to mere milliseconds. Users can run muiltiple metrics simultaneously, yielding insights into the AI's behavior more efficiently than with standard LLM judges, which are often costly and time-consuming. The significance of jevals lies in its ability to provide precise evaluations without the inherent unpredictability of LLM judges, enhancing assessment accuracy for AI agents performing tasks. Jevals uses typed questions to independently evaluate components like tool selection, answer relevancy, and adherence to given scopes, yielding a more reliable evaluation framework. Additionally, with support for local models like Kev and Laya, organizations can adopt jevals without incurring hefty costs associated with cloud-based LLMs. This tool thus represents a substantial advancement in agent performance evaluation and could redefine how AI systems are tested and refined in real-world applications.