A local RAG assistant for a Romanian government agency, built on an 8GB laptop (arxiv.org)

A Romanian government agency, AFIR, has developed RAGAL, a local retrieval-augmented assistant designed for technical support, all while adhering to strict constraints such as zero data egress and operating on an 8GB consumer laptop. This development is crucial because public institutions often deal with sensitive documents that must remain onsite, making traditional cloud-based AI solutions impractical. RAGAL leverages a corpus of approximately 25,000 Romanian-language documents and emphasizes retrieval engineering and retriever fine-tuning, achieving a significant performance boost in internal evaluations—from 62% to 81% accuracy. Key technical innovations include hybrid dense-sparse retrieval with intent routing and fine-tuning a specialized embedder, which improved recall metrics notably. Interestingly, the project revealed that fine-tuning on a narrow domain can negatively impact performance on other areas, prompting a novel use of locally generated queries to enhance results. Additionally, the team developed a unique SQL hallucination mitigation method through "anchor distillation." By sharing their sanitized pipeline scripts, the RAGAL initiative provides a valuable blueprint for other institutions grappling with similar data-locality challenges, showcasing how robust AI tools can be developed under limited resources.