Show HN: Rubric – test what your LLM agent did, not just what it said (github.com)

A new tool called Rubric has been introduced to enhance the testing of behavior in LLM (Large Language Model) agents. Unlike traditional evaluation methods that only assess the output of an agent, Rubric focuses on the entire process of agent behavior, including the tools used, the arguments passed, error tracking, latency, and reasoning traces. This approach allows developers to identify regressions in Continuous Integration (CI) before deploying their models, ensuring more reliable performance in real-world applications. Rubric operates without any dependencies, making it fully local and easy to integrate into existing workflows. Users can evaluate their agents simply by installing the package and writing straightforward test cases. Key features include tool call accuracy checks, latency metrics, and safety compliance assessments, all of which highlight potential issues that could compromise an agent’s performance. By enabling automatic evaluations and providing detailed reports, Rubric not only streamlines the testing process but also empowers developers to catch hidden bugs that traditional methods might overlook, making it a significant addition to the AI/ML community's toolkit.