Show HN: LLMhop – A tiny, stateless router for LLMs with a NixOS module (github.com)

LLMhop, a minimalist, stateless HTTP router designed for OpenAI-compatible LLM inference backends, has been introduced to streamline model requests. It allows a single endpoint to handle inference requests by examining the model field in incoming requests and reverse-proxying them to the appropriate backend, making it especially useful for single-model servers like vLLM and sglang. With zero external dependencies and a static binary format, LLMhop aims to simplify the deployment of various LLMs, working seamlessly with self-hosted or remote backends including TabbyAPI and OpenRouter. This tool is significant for the AI/ML community as it enhances the efficiency of model management, catering to both small-scale implementations and larger, multi-model frameworks. Key features include guarded request gating using bearer tokens, configurable per-model headers, and a cap on request body sizes to protect server memory. Additionally, it supports advanced configurations through a hardened NixOS module and can run inference servers directly, facilitating a more secure and flexible environment for deploying AI models. Overall, LLMhop provides a robust infrastructure solution to meet the growing demands of LLM deployment and management.