Self Hosted LLM Gateway and Feature Rich Chat UI with RBAC (github.com)

A new self-hosted LLM Gateway has been announced, which includes a feature-rich chat UI and supports role-based access control (RBAC). This authenticated, OpenAI-compatible reverse proxy can route LLM requests across multiple backends using an OIDC login system, per-user API tokens, and server-side tools. It operates from a single binary with SQLite for state management, making it easy to deploy and utilize. Key features include a built-in chat interface, multi-backend routing, real-time model discovery, and robust server-side tools that permit the model to execute operations like web searches and document rendering during completion. This development is significant for the AI/ML community as it offers a comprehensive solution for teams aiming to leverage LLMs securely and efficiently. The ability to manage user permissions through RBAC while integrating various backend tools enhances customization and control. Technical features like scheduled actions, model aliases, and seamless integration with external services (e.g., Google Workspace, GitHub) highlight its versatility. Additionally, the gateway's support for dynamic token management and user-specific connectors fosters a personalized experience and efficient resource use, positioning it as a compelling option for organizations utilizing AI-driven solutions.