AI Discovery Index (mojodojo.io)

The recently launched AI Discovery Index provides valuable insights into the speed and efficiency of various web crawlers, including both traditional search engines like Bing and Google, as well as AI-specific crawlers such as ClaudeBot and PerplexityBot. The data reveals that 51% of pages were first crawled by Bingbot, followed closely by ClaudeBot at 49%. Remarkably, 40% of new pages received a visit from an AI crawler within just one day of publishing, underscoring the increasing significance of AI in how information is discovered and indexed online. This index is significant for the AI/ML community as it highlights the competitive landscape of web crawling, offering a glimpse into how quickly and effectively AI tools can analyze new content. With median response times averaging around 14 hours for both Bingbot and ClaudeBot, these metrics suggest that AI crawlers are becoming essential in the race for information retrieval and indexing. The findings also pose implications for web content creators and SEO strategies, as understanding crawler behavior can influence how content is optimized and published, ultimately affecting visibility and engagement.