Crawlerpeek – check which AI crawlers your site lets in (CLI, MCP) (github.com)

Crawlerpeek, a newly launched tool, allows website owners to check which AI crawlers can access their site effectively. This command-line interface (CLI) and Model Context Protocol (MCP) tool evaluates how well various bots, including GPTBot, ClaudeBot, and Googlebot, interact with a site's robots.txt file. It provides detailed feedback on whether access is granted, the HTTP status of fetched pages, and any challenges encountered, such as block pages. Crawlerpeek also generates a custom robots.txt to optimize AI crawler access, making it easier for developers to manage web scraping and search engine indexing. The significance of Crawlerpeek lies in its potential to enhance the transparency and control website owners have over AI crawler interactions. Understanding how different bots respond to their site can help optimize content visibility and performance in search engines. With no runtime dependencies and support for both ESM and CommonJS modules, developers can easily integrate Crawlerpeek into their CI/CD pipelines. It delivers critical insights into bot behavior while highlighting potential issues with accessibility, thereby enabling webmasters to fine-tune their site’s AI accessibility strategy efficiently.