Show HN: glide – LLM cascade proxy, auto-switches models before timeout (github.com)

A new tool called "glide" has been introduced, designed to enhance the performance of AI applications by automatically switching to faster models when the primary model is slow. This tool acts as a proxy between AI agents and the Anthropic API, drastically reducing the response time from potentially 15 seconds to about 2 seconds. The innovative LLM Request Cascade Pattern employed by glide utilizes a tiered model approach, where it monitors the time-to-first-token (TTFT) for each model and can proactively skip models that exceed their latency budgets. The significance of glide lies in its ability to maintain responsiveness in AI coding tasks, helping developers avoid disruptions during critical work. Glide tracks the rolling p95 TTFT of various models, dynamically routing requests to ensure the quickest response is always prioritized. Featuring seamless integration with multiple AI service providers like Anthropic and OpenAI, glide represents a substantial leap towards creating reliable, real-time AI systems, helping to keep AI agents operational even under peak loads or during individual model slowdowns. This flexibility and efficiency can greatly benefit developers relying on AI for coding assistance and other time-sensitive applications.