A 3.5 MB C++ engine for deterministic RAG deduplication hitting 30 GB/s (github.com)

A new lightweight C++ engine for deterministic deduplication in retrieval-augmented generation (RAG) systems has been released, boasting an impressive throughput of 30 GB/s. The local-first solution, available under an MIT license, allows users to significantly reduce duplicate input data in large language model (LLM) contexts, achieving up to 71% deduplication in RAG pipelines. This targeted efficiency addresses the issue of wasted data inputs, potentially lowering costs for developers and organizations leveraging LLMs. The community edition includes various integrations, such as a MCP server and a VSCode extension, while a more robust enterprise version, featuring a multi-threaded and lock-free architecture, remains proprietary. The community tool simplifies installation and usage—highlighted by its straightforward command structure—and offers a shared ledger to track savings. This development signifies a critical step toward enhancing the performance and cost-effectiveness of AI-driven applications, as it provides users with essential tools to streamline their workflows while protecting against data interception by third-party services.