Show HN: Mantis, A self-hosted LLM gateway (github.com)

Mantis has launched as an open-source, self-hosted LLM gateway designed to streamline the development of applications utilizing multiple language models. This platform allows teams to interact with various LLMs through a single, stable chat-completions API, enhancing operational management by centralizing routing, caching, failover behavior, and guardrails. Mantis is particularly significant for small teams aiming to maintain control over their infrastructure and data while leveraging the power of large language models. Key features include configurable routing capabilities based on metadata and model aliases, exact and semantic response caching for efficiency, and AWS Bedrock guardrails to ensure compliance with data privacy standards. The gateway's deployment is AWS-native, utilizing tools like Terraform and ECS Fargate, which simplifies infrastructure management. With Mantis, teams can develop multi-LLM applications more reliably while keeping their codebase clean and focused. This simplification of the model selection and request handling process positions Mantis as a valuable asset for developers in the AI and ML community seeking to enhance their application's functionality without the complexity of individual model integrations.