Show HN: Cascade – A bare-metal C++ proxy that cuts LLM API bills by 70% (cascade-router.github.io)

A team of developers has introduced Cascade, a bare-metal C++ proxy designed to reduce costs associated with large language model (LLM) API usage by up to 70%. This innovative solution rapidly predicts prompt complexity in just 4.59 milliseconds and intelligently routes requests to the most cost-effective models, minimizing API expenses without introducing latency. The system efficiently handles tokenization, ONNX embedding, and ML predictions in an end-to-end manner, allowing businesses to optimize their workloads according to cost and performance. Cascade's intelligent routing can automatically manage simpler tasks—like extraction and classification—by directing them to faster, less expensive models instead of more resource-intensive ones. For instance, it estimates potential monthly savings of $28,125 and an impressive annual savings of $337,500, assuming a 60% cost differential between fast and frontier models. Additionally, Cascade supports state preservation, ensuring that if a simpler model fails validation, the requests are escalated to higher-performance models without any disruption. With an open-source option available for self-hosting and enterprise features for custom configurations, Cascade presents a significant opportunity for companies looking to enhance their AI infrastructure while significantly cutting costs.