Cloudflare hid my site from AI. I sell AI visibility (foundash-ai.github.io)

A website owner specializing in providing AI visibility services discovered that their site was unintentionally hidden from AI crawlers due to a Cloudflare setting. This configuration blocked access to their site for several prominent AI training bots while still allowing access to search-time agents. The owner identified issues with their robots.txt file and a parser bug that previously masked their site's "invisibility" to these AI tools. Upon re-scanning, they found that about 5% of local business websites were blocking AI crawlers, but many of those did not explicitly communicate their opt-out status, obscuring the true extent of the issue. This discovery is significant for the AI/ML community as it highlights a gap in how online visibility is maintained for local businesses in the context of evolving AI search technologies. The findings indicate that many local businesses might unknowingly restrict access to their sites, potentially debilitating their online recommendations and visibility to AI assistants. The study underlines the importance of correctly managing robots.txt settings and understanding the differences between training crawlers and search-time agents, encouraging website owners to reevaluate their configurations to ensure they remain recommendable in the AI landscape.