Show HN: Suture – a reverse proxy that repairs truncated JSON in LLM streams (github.com)

Suture, a newly announced ultra-low-latency reverse proxy, addresses the common issue of truncated JSON in streaming responses from large language models (LLMs). When upstream streams are interrupted—due to limitations like max tokens or dropped connections—applications often encounter JSONDecodeErrors. Suture operates as an intermediary between the application and LLM providers, automatically repairing truncated JSON outputs on the fly without adding latency or buffering requirements, ensuring that clients receive valid JSON responses seamlessly. This tool is particularly significant for developers working with popular LLMs from providers like OpenAI, Anthropic, Google Vertex AI, and AWS Bedrock, which frequently generate structured outputs that can get cut off. It employs a three-layer architecture that includes a byte-level JSON repair engine and an incremental SSE parser, allowing it to handle streamed and compressed content. Suture does not store any credentials, maintaining security while supporting multiple streaming formats. The implementation can be easily integrated into existing applications, making it a valuable asset for enhancing the robustness of LLM-based services in the AI/ML community.