Show HN: OmnisBench, a re-gradable, open LLM routing benchmark on fresh tasks (github.com)

OmnisBench has introduced a novel, open-source benchmarking tool designed to evaluate the routing efficiency of large language models (LLMs) against real-time, fresh tasks. Unlike previous benchmarks, OmnisBench continuously updates its routing policies to reflect current costs and tasks that have emerged after the models' training cutoffs. This live framework allows researchers to assess how well different routing strategies can optimize the quality-to-cost ratio of LLMs when tackling previously unseen problems, with a notable example revealing that an ideal routing strategy could achieve a 93.3% success rate at significantly lower costs than a default model selection. The significance of OmnisBench in the AI/ML community lies in its commitment to providing a clear and reproducible method for assessing LLM performance under practical conditions, devoid of contamination from prior training data. It incorporates a systematic approach to classify tasks based on freshness, ensuring that routing capabilities are accurately evaluated. The tool also emphasizes the critical gaps between theoretical efficiency and practical application, paving the way for future research and enhancements in model routing strategies. With its focus on real-time benchmarking and continuous updates, OmnisBench could greatly influence how AI practitioners approach model evaluation and deployment in diverse use cases.