Show HN: Doc-scraper, offline searchable docs corpora for coding agents (Go) (github.com)

A new tool called "doc-scraper" has been launched, designed as a concurrent and configurable web crawler specifically for extracting technical documentation from websites. Developed in Go, this tool automates the process of gathering, cleaning, and formatting technical documentation into Markdown, making it easier for Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems to ingest structured information. Users can define crawling settings through a `config.yaml` file to effectively navigate website structures and extract relevant text content based on CSS selectors. The significance of this project lies in its ability to streamline the often tedious task of compiling comprehensive datasets from web-based documentation. The tool maintains the original site hierarchy for context, utilizes Go's concurrency for efficient parallel processing, and includes production-ready features like resumable crawling, rate limiting, and graceful error handling. The output can be organized for local access, making it accessible for further processing, and even allows for offline full-text searching over the crawled documents, enhancing its utility for AI-driven applications in software development and documentation.