Online Learning for Cost-Efficient LLM Routing (builders.ramp.com)

Ramp has developed an internal LLM gateway that processes trillions of tokens daily, optimizing AI usage for both internal development and external products. By implementing a dynamic failure-aware routing strategy, Ramp has successfully reduced AI spending by over 25% while simultaneously decreasing error rates. This innovation arose from analyzing latency patterns and incorporating real-time data into their routing decisions, allowing for a more efficient allocation of resources across different service tiers. The strategy leverages an exponentially-weighted moving average (EWMA) to monitor genuine provider-side failures and utilizes Thompson Sampling to evaluate latency distributions in real-time. This adaptive approach allows Ramp to dynamically choose the most cost-effective model based on latency and failure probabilities. The initial implementation for LLM reranking demonstrated immediate results, including substantial cost savings and maintained performance levels, with further success seen in streaming applications achieving 30% savings. Ramp's focus on optimizing AI spend through this centralized gateway positions them as a leader in efficiency within the AI/ML community.