Show HN: ShanHaiGateway – multi-upstream LLM gateway in Go (github.com)

A new project titled ShanHaiGateway has emerged, designed as a multi-upstream LLM (Large Language Model) gateway implementing an OpenAI-compatible endpoint using Go. This tool integrates multiple model providers, allowing users to manage requests through sophisticated features such as weighted round-robin (SWRR), quota-based index cooling, and task-level session affinity. These capabilities are particularly significant as they address common challenges of managing multiple LLM providers, such as handling errors more intelligently, maintaining session continuity to improve cache hit rates, and providing a seamless integration without requiring extensive configuration. ShanHaiGateway shows promise in cost reduction by leveraging a mix of free and paid model pools, which collectively support massive token throughput (e.g., 200 billion tokens monthly for approximately ¥170), in stark contrast to the estimated ¥26,600 if billed at standard directory rates. Its architecture emphasizes operational resilience with features like detailed error classification and cooldown strategies to mitigate failures. This project not only highlights practical operational rules developed from real-world experiences but also makes advanced LLM technologies more accessible and affordable for developers and businesses, reinforcing the community’s ongoing journey in optimizing AI/ML deployment strategies.