Show HN: CRoM – Context Rot Mitigation System for RAG-Based LLMs (github.com)

CRoM (Context Rot Mitigation)-EfficientLLM is a new open-source Python toolkit designed to enhance the performance of Retrieval-Augmented Generation (RAG) pipelines by optimizing how text chunks are selected, ranked, and managed within the limited context windows of large language models (LLMs). By intelligently packing the most relevant content into a defined token budget, CRoM helps minimize semantic drift—where model responses gradually deviate in topic or quality—addressing a key challenge in maintaining coherence over long interactions. Technically, CRoM features a hybrid reranker merging sparse TF-IDF and dense Sentence-Transformer embeddings to improve document prioritization, alongside a drift estimator that uses L2 or cosine distance with exponential smoothing to track changes in response semantics. It supports extensible plugins for advanced reranking (FlashRank), compression (LLMLingua), and drift analysis (Evidently), while offering Prometheus metrics for real-time monitoring in production. Comprehensive benchmarking tools, including a command-line interface for evaluating end-to-end pipelines under different token budgets, enable developers to rigorously analyze trade-offs between quality and resource constraints. This project is significant as it provides a practical, modular solution for developers grappling with context window limitations in LLM-powered applications, especially in domains requiring the integration of external knowledge. By improving relevance and reducing information drift in RAG setups, CRoM stands to advance the reliability and efficiency of AI systems that rely on dynamic retrieval and generation.