Show HN: Millwright – Rust-based, self-hosted LLM router (github.com)

Millwright, an open-source LLM router written in Rust, has been launched to facilitate efficient routing of AI model requests while managing costs, cache affinity, and policy compliance. This self-hosted solution enables users to route requests from AI applications to various model providers, including OpenAI and Anthropic, by defining models as "cheap," "mid," or "frontier." Millwright intelligently selects the most cost-effective route for each request while maintaining session continuity to maximize prompt-cache reuse. It preserves provider affinity, which is crucial for minimizing the costs that arise from frequent changes in model providers. The significance of Millwright for the AI/ML community lies in its ability to streamline the process of managing AI model interactions and costs without altering application code. With features such as traffic classification, concurrent lanes for different model roles, and transparent spend analytics, it empowers developers to optimize their AI workloads effectively. Additionally, its robust architecture allows for seamless switching between providers while keeping sensitive credentials secure within the user's infrastructure. Millwright's design promotes transparency and introspection, offering insights into routing decisions and usage patterns, making it a valuable tool for organizations looking to enhance the efficiency of their AI operations.