Show HN: CLI for crawling documentation sites into Markdown with defuddle (github.com)

A new command-line interface (CLI) tool called docrawl has been introduced, designed for efficiently crawling documentation sites and converting their content into Markdown format. Built using Node.js, docrawl is compatible with various static and server-rendered documentation platforms, including Docusaurus and MkDocs, and allows users to convert web documentation into a format suitable for local storage, knowledge bases, or integrating into retrieval-augmented generation (RAG) pipelines. The CLI operates without a browser or JavaScript execution, which enhances its efficiency and usability across different environments. This tool is significant for the AI/ML community as it streamlines the process of gathering and archiving documentation, ensuring that developers and researchers can easily access structured content for training language models or building applications from concise documentation. Key features include customizable crawl settings—such as limiting the number of pages processed, setting crawl depth, and outputting either individual Markdown files or a single merged file with a manifest. However, it currently does not support content that requires JavaScript rendering or user authentication, indicating specific limitations for its use in dynamically generated sites. Overall, docrawl stands to simplify the documentation management workflow for AI developers.