Show HN: CockroachCrawler–1 package for crawling,browsers,PDF,extraction,and MCP (github.com)

Cockroach Crawler has been introduced as an open-source AI web crawler that empowers agents with the ability to crawl entire websites, render JavaScript, and extract structured data cleanly without needing unrestricted access to the internet. Built using TypeScript, this tool is particularly significant for AI/ML applications involving research, documentation indexing, and content quality assurance. It consolidates various crawling methods—such as breadth-first and depth-first search—within a single package that supports generating clean outputs in Markdown, JSON, or JSONL formats while adhering to strict provenance guidelines. The tool is designed specifically for environments that require governed operations, providing an array of features like bounded crawling, URL mapping, and PDF parsing—all while ensuring compliance with site policies and regulations. Cockroach Crawler does not bypass security measures, focusing instead on ethical data extraction from explicitly allowed public sources. Key functionalities include the ability to capture content hashes and metadata, ensuring transparency and auditability. This makes it ideal for researchers and developers who need reliable access to structured data without compromising on security aspects. With its emphasis on local deployment and open-source accessibility, Cockroach Crawler is positioned to enhance the efficiency and integrity of data collection in AI workflows.