Show HN: Relay – a self-hosted LLM gateway with smart routing and request pacing (github.com)

A new self-hosted gateway called Relay was launched on Product Hunt, designed to streamline interactions between applications and various AI models. This tool enables developers to connect an extensive range of AI providers—including OpenAI, Anthropic, and local models—through a single API, enhancing backend flexibility and system efficiency. Key features include smart routing capabilities that ensure optimal model usage, request pacing to handle sudden traffic spikes while adhering to provider rate limits, and real-time usage tracking to monitor costs and performance. The significance of Relay lies in its ability to centralize model management, allowing developers to avoid hard-coding specific models into applications and facilitating easier experimentation with different AI systems. With functionalities like automatic request categorization, fallback model strategies, and built-in security for managing sensitive credentials, Relay provides a robust infrastructure for AI-driven applications. This gateway not only simplifies deployment but also empowers users to optimize their AI operations, making it an essential tool for the evolving AI/ML landscape.