RoBC – LLM Routing on Bayesian Clustering (github.com)

RoBC, an innovative online learning LLM router, has been introduced to meet the challenges of dynamic production environments where model quality frequently shifts. This router employs Thompson Sampling combined with semantic clustering to adapt in real time—so it does not require retraining as models change or new ones are released. Unlike static routers that become outdated or ineffective with quality drift, RoBC improves its routing decisions with each request, ensuring ongoing optimization based on current model performance. The significance of RoBC lies in its capacity to enhance model routing efficiency—demonstrated through evaluations against the static RoRF router. In scenarios with quality drift or the introduction of new models, RoBC outperformed RoRF by 15.2% and 19.5%, respectively, due to its adaptive learning capabilities and low overhead (~1ms for routing decisions). Key components include a Cluster Manager for prompt assignment and a Bayesian Posterior Manager, all designed to improve routing decisions without the complexities of a traditional retraining pipeline. RoBC thus represents a leap forward for the AI/ML community, particularly in production settings where adaptability and immediacy are crucial.