Show HN: I stopped using an LLM gateway and put rate-limits/fallback in-process (github.com)

A new open-source framework called VernLLM has been introduced to simplify interactions with various large language models (LLMs) from providers like OpenAI, Anthropic, and Gemini. This framework is remarkable for the AI/ML community as it allows developers to consolidate communication with multiple LLMs into a single interface while incorporating essential features such as retries, rate limits, circuit breaking, and middleware—all running in-process, which minimizes latency associated with network hops. The code snippet included demonstrates the setup of the framework, highlighting its flexibility to seamlessly switch between different LLMs and set customized rules for usage metrics. The significance of VernLLM lies in its resilience and observability in handling LLM calls, enabling better control over model interactions. By offering a structured approach to implement rate limits (e.g., 500 requests per minute), caching mechanisms, and fallback options for service interruptions, it empowers developers to create more robust AI applications. This design not only enhances performance but also ensures a smoother user experience when working with multilingual models. Overall, VernLLM represents a substantial leap forward in making LLM functionality more accessible and efficient for a diverse range of AI applications.