Show HN: Directory of 28 AI crawlers – robots.txt rules, IP ranges, open data (geoprompttracker.com)

A newly launched directory detailing 28 AI web crawlers has been made available, providing essential information on the various bots that harvest data for major AI models like ChatGPT, Claude, and Google’s Gemini. Each entry in the directory outlines the crawlers' functionalities, their compliance with robots.txt rules, and includes customizable scripts for website owners to manage access effectively. This resource is refreshed monthly and is especially critical as it enables developers and content creators to make informed decisions about what data their sites share with AI systems. The significance of this directory stems from increasing concerns over data privacy and responsible AI training practices. By clarifying which crawlers respect or ignore robots.txt directives, the directory offers a vital tool for stakeholders in the AI/ML community to better navigate interactions with these data-gathering agents. Furthermore, the directory includes a robot.txt Generator and an AI Bot Log Analyzer, which allow users to assess and tailor their interactions with the bots, thereby empowering them to strike a balance between advancing AI research and protecting their digital assets.