Trace-Based Adaptive Cost-Efficient Routing (github.com)

A new framework called TRACER has been introduced to optimize the routing of classification tasks in large language model (LLM) pipelines. Unlike traditional approaches that rely on LLMs for every input, TRACER leverages a lightweight traditional machine learning (ML) model to handle most typical requests, deferring only ambiguous cases to the LLM. By learning the decision boundaries from the LLM's classification traces, TRACER can maximize efficiency— routing over 90% of classification calls to the faster, cost-effective ML models, thereby significantly reducing operational costs. This advancement is particularly significant for the AI/ML community because it not only enhances the scalability of LLM applications but also offers a self-improving mechanism. As TRACER receives more input, it continually refines its classification capabilities, increasing coverage and maintaining accuracy with formal parity guarantees against the LLM. The process of fitting and updating the routing policy is straightforward, allowing for seamless integration into existing JavaScript-based ML pipelines. With projected annual savings of over $300,000 for 10,000 queries per day, TRACER is a game-changer for companies reliant on LLMs, improving both performance and cost-effectiveness in practical applications.