Stop fine-tuning LLMs for docs, use RAG (intlayer.org)

The author built and open-sourced a ready-to-run RAG (Retrieval-Augmented Generation) documentation assistant—Next.js + React UI, Node/TypeScript backend, and an embeddable docs-processing package—that turns static Markdown docs into a searchable, chat-style help system. It includes chunking, embeddings, cosine-similarity retrieval, and prompt augmentation for ChatGPT, plus query logging so product teams can surface missing docs, recurring pain points, and feature requests. The boilerplate ships with editable Tailwind UI, SSE streaming, and a low-cost footprint (embedding ~200 docs ≈ €1–2; ~300 monthly chats typically under $10 on OpenAI). Technically, the pipeline chunks docs (example: ~500-token chunks with ~100-token overlap), generates vectors with OpenAI’s text-embedding-3-large (or any embedding model), stores indexes (prototype used embeddings.json; production should use Chroma/Qdrant/Pinecone/FAISS), and retrieves top-N chunks via cosine similarity. Retrieved chunks are injected into the system prompt for gpt-4o-latest/gpt-4-turbo to produce grounded answers (gpt-5 was too latency-heavy). Practical notes: chunk size/overlap and number of retrieved chunks need tuning, embeddings must be re-generated when docs change, and vector stores scale much better than JSON. Beyond search accuracy, the big payoff is the feedback loop—RAG doubles as user research/product-intelligence by logging real queries and surfacing gaps and new use cases.