Show HN: Sturnus, a latency-based LLM load balancer (github.com)

Sturnus has been introduced as a latency-based load balancer for large language models (LLMs), designed to optimize the routing of requests across multiple LLM providers automatically. As a lightweight sidecar application, it operates alongside existing applications without requiring additional infrastructure, simplifying deployment to a single static binary. By dynamically directing traffic to the fastest and most available provider, Sturnus ensures consistent performance, which is crucial for production environments where latency and availability can significantly impact user experience. This tool is particularly significant for the AI/ML community as it minimizes the risk of relying on a single LLM provider, thereby enhancing resilience and efficiency. Sturnus exposes an OpenAI-compatible API, making it easy for developers to swap API endpoints with minimal changes. Its technical capabilities include memory-bounded request handling, transparent passthrough for raw request/response data, session affinity for multi-turn interactions, and automatic recovery for previously slower providers. Moreover, it includes support for Vertex AI and can seamlessly integrate with existing client SDKs while promoting a stateless, auditable, and low-surface-area architecture that prioritizes speed without compromising quality.