Show HN:Lumisift – improves data retention in RAG from ~40% to 87% (github.com)

Lumisift, a new tool developed by Saeed Moradtalab, significantly enhances data retention in Retrieval-Augmented Generation (RAG) systems, boosting it from an alarming 40% to 87%. Traditional RAG systems often select context based on semantic similarity, leading to the loss of critical scientific data—64% of numerical facts and 61% of comparative claims, as evidenced by analysis of 1,077 PubMed articles. Lumisift addresses this gap by inserting an information density detection step that prioritizes data-rich paragraphs, ensuring that AI systems deliver precise experimental results instead of vague approximations. This innovation is particularly vital for the AI/ML community, especially in fields such as pharmaceuticals and clinical research, where missing data points like IC50 values or p-values can have serious ramifications, including flawed research or compliance failures. By effectively enhancing the retrieval process without needing additional models or cloud services, Lumisift retains essential measurements and statistical evidence, leading to more reliable AI-generated outputs. Notably, it outperformed several baseline retrieval methods, robustly demonstrating its ability to preserve critical data while maintaining a user-friendly, locally runnable architecture.