Show HN: Keep large tool output out of LLM context: 3x accuracy 95% fewer tokens (github.com)

Sift, a new reliability layer for managing large tool output in AI applications, has been launched, significantly enhancing the handling of complex data within language models. It enables the persistent storage of JSON data and allows for efficient querying using Python, thereby optimizing for both accuracy and token usage. In benchmarking tests, Sift achieved an impressive accuracy rate of 99% on factual questions while drastically reducing input tokens from over 10 million to under half a million. This represents a remarkable +66% improvement in accuracy compared to traditional methods. This development is significant for the AI/ML community as it provides a structured approach to manage untrusted tool outputs, addressing challenges like context overflow and parsing inconsistencies across different servers. By separating large outputs from the model's input context and ensuring the integrity of the data with features like schema-aware references and secure handling of sensitive information, Sift enhances reproducibility and reliability in AI processes. Its explicit pagination capabilities further streamline data retrieval and analysis, making it a critical tool for developers working with complex datasets.