Open-ultra: a self-training LLM routing proxy (github.com)

Open-ultra has introduced a self-training LLM routing proxy designed to optimize the performance and cost of coding agents. This innovative system allows users to run traffic through cheaper models while maintaining the output quality of their primary "frontier" model. By leveraging a hidden shadow lane that tests cheaper options against real work submissions, Open-ultra dynamically routes requests to models that have demonstrated comparable performance. This enables users to capitalize on cost savings without sacrificing quality on tasks where the cheaper models excel. The significance of Open-ultra lies in its potential to make AI-driven coding more accessible and economical. It continuously calibrates to real-time traffic, eliminating the need for synthetic benchmarks, and assesses models based on live performance. The tool allows for granular control over model selection, with a focus on real-world efficacy, ensuring that even under heavy load, users benefit from the lowest available rates while safeguarding the quality of their projects. With a user-friendly CLI and seamless integration with existing systems, Open-ultra positions itself as a powerful tool for developers looking to optimize their AI workflows.