Show HN: LLM-schema-guard – Rust proxy enforcing JSON schemas on LLM outputs (github.com)

A new high-performance HTTP proxy named **llm-schema-guard** has been introduced, designed to validate outputs from Large Language Models (LLMs) against specified JSON schemas. Developed in Rust, this lightweight tool serves as a drop-in replacement for OpenAI-compatible endpoints and features automatic schema validation, retry logic for failed validations, and integration with Prometheus for detailed metrics. Significant capabilities include strict and permissive modes for handling validation errors, support for streaming responses, and Docker readiness for easy deployment. The introduction of llm-schema-guard is particularly important for the AI/ML community as it provides a robust mechanism to ensure that model responses adhere to defined schemas, which can help developers build more reliable and predictable applications. Key features, such as customizable retry strategies with exponential backoff, capability for in-memory caching to improve performance, and detailed observability metrics, enhance its utility in production environments. This tool addresses common challenges in LLM deployments, allowing for better quality assurance and streamlined interaction between clients and LLMs.