Show HN: A model-routing benchmark – the routers optimize the wrong axis (github.com)

A new model-routing benchmark and tutorial series has been introduced, focusing on which AI model should address specific queries related to the 2023 8th-Edition Florida Building Code and local amendments. The benchmark evaluates models across three tiers—cold, grounded, and various configurations—utilizing a deterministic citation-match metric against a set of 45 real questions. The results reveal how well different models perform with and without contextual grounding, highlighting significant discrepancies in accuracy, particularly with cold models, which performed unsatisfactorily. The findings indicate that grounding—supplying the model with the necessary context to enhance its answers—is crucial for effective model routing. For instance, while the cold model Claude Opus 4.8 had only a 26.5% correctness rate, the grounded phi3 model achieved 52.5%, demonstrating that infusing relevant context can significantly boost performance. This benchmarking emphasizes that model selection is not the only factor for success; grounding must be prioritized as a fundamental part of the query-response process in AI development. Furthermore, the free and open-source nature of the benchmark allows developers to adapt it for various applications, underscoring the ongoing need for reliable, context-aware AI models in practical scenarios.