Show HN: ArXiv Scholar – An Open-Source RAG System for AI Research Papers (github.com)

ArXiv Scholar has launched as an innovative, open-source Retrieval-Augmented Generation (RAG) system specifically designed for AI Engineering research, capable of efficiently processing and retrieving academic papers from arXiv. This end-to-end pipeline ingests PDFs, parsing and chunking them into a hybrid vector database that facilitates rapid semantic searches. Currently, it indexes around 5,600 papers on Qdrant Cloud and employs a streaming API for real-time interaction, showcasing its potential for enhancing research accessibility and usability. This system's significance lies in its comprehensive architecture, built without high-level abstractions, which provides full control and transparency in operation. Key technical features include intelligent query routing that classifies user queries based on complexity and a layout-aware PDF parsing method that preserves the semantic structures of academic documents. Furthermore, it integrates dense and sparse vector embeddings for superior search performance and employs a dynamic compute budgeting strategy to manage resource allocation efficiently. Such advancements position ArXiv Scholar as a vital tool for researchers, enhancing their ability to navigate and synthesize information from vast amounts of scientific literature.