An LLM API gateway with budget-limited tokens for your LLM API keys (github.com)

A new LLM API gateway has been announced that enhances security and budget management for AI/ML developers. The gateway aims to keep real API keys from being exposed to clients by using opaque tokens that are tied to specific API providers, with built-in features such as expiration times, IP allowlisting, and spending budgets. The gateway operates on Cloudflare, utilizing its Workers and D1 database to handle requests securely. These features help mitigate risks associated with key leakage while allowing developers to better manage their API costs. This implementation is significant as it provides a way to customize access controls and budget limits for API usage without sacrificing convenience. However, there are caveats: budget control is not strictly enforced, as a single high-cost request could exceed the defined budget, and reliability in budget tracking can be an issue if errors occur. Prospective enhancements include better monitoring and the potential for dynamic cost controls per request. Overall, this gateway offers a promising solution to some of the current challenges for developers interacting with multiple LLM APIs.