Show HN: Relay – a self-hosted LLM gateway with eval-gated routing (github.com)

A new self-hosted gateway and model router called Relay has been introduced, allowing users to interface with various language models (LLMs) like OpenAI, Anthropic, and others using a single static Go binary. Relay’s unique feature is its ability to implement eval-gated routing, enabling smarter decision-making about which model to use based on the complexity of the task at hand. This functionality allows users to define routing policies from static aliases to dynamically learned strategies while maintaining strict privacy standards, as it collects no telemetry data. This development is significant for the AI/ML community as it offers a way to seamlessly switch between different model providers with minimal configuration and optimal performance. The architecture supports smart routing based on user-defined criteria and logs interactions for analysis, improving routes over time without sending prompts to third parties. With performance metrics indicating minimal overhead, Relay is positioned as a highly efficient solution for leveraging multiple LLMs while preserving data privacy, making it a valuable tool for developers and businesses looking to enhance their AI capabilities.