Running RAG with a (small) language model in the browser (bart.degoe.de)

MicroLLM lab has launched a novel approach to running small language models (ranging from 25 to 360 million parameters) directly in the browser, without the need for servers, APIs, or accounts. This setup incorporates retrieval-augmented generation (RAG) functionality, allowing users to input a question and receive concise summaries from related blog passages using a lightweight model. By leveraging WebGPU or WebAssembly, the browser computes responses with high efficiency, ensuring that user data remains private and all processing takes place locally. However, testing revealed that smaller models often struggle with accuracy, frequently generating responses that lacked fidelity to the source material. This project is significant for the AI/ML community as it demonstrates the potential of running effective language models on personal devices, opening doors to increased privacy and accessibility in AI usage. The findings indicate that while small models are cost-effective and agile, they still face challenges with reliability and relevance, particularly in summarizing complex content accurately. The author plans to refine model performance through fine-tuning and explore ways to enhance search capabilities using loaded models, highlighting an ongoing evolution in the user-friendly application of AI technologies.