Probelock – lockfile for LLM tool calling (github.com)

Probelock, a new capability lockfile for local large language models (LLMs), has been announced as a tool for tracking model performance across various tool-calling and output checks. Designed to enhance continuous integration (CI) practices, Probelock records a model's performance metrics and automatically signals when a change in model, quantization, or runtime configuration results in degraded performance. For example, transitioning from a llama model at Q8_0 quantization to Q4_K_M showed substantial capability regressions in key areas like argument validity and tool discrimination, revealing the need for vigilant model management. The significance of Probelock lies in its approach to benchmarking and testing. Unlike traditional absolute leaderboards, Probelock allows developers to compare their models against a baseline within their specific configurations, eliminating issues with hardware-dependent benchmarks. The framework operates without LLM judgment, relying solely on deterministic scoring through JSON-schema validation and tool checks. This innovative methodology not only aids in identifying significant regressions but also streamlines the testing process, making it easier for AI/ML practitioners to maintain model quality and performance over time.