Measure agent-traffic for any site (github.com)

Mintlify has launched an innovative tool called "agent-traffic" that allows developers to distinguish between various types of web traffic—specifically, to identify requests made by AI crawlers, bots, and human users. With increasing traffic to documentation hosted on Mintlify from AI tools like Claude Code and ChatGPT, the need arose to accurately assess traffic sources and ensure that developers could differentiate genuine requests from impersonators. The new tool utilizes a curated dataset of user agents, headers, and verification sources, along with a simple classification system, to provide insights into the nature of incoming requests. This development is significant for the AI/ML community as it enhances transparency in web traffic analysis, allowing companies to understand the interactions between their documentation and AI tools more effectively. By open-sourcing the agent-traffic package, Mintlify empowers developers to implement this classification in their own applications without having to tackle the complexities of traffic analysis from scratch. The system runs on various platforms and supports detailed verification against published IP ranges, ensuring a reliable way to identify genuine agents and bots, thus safeguarding against data scraping and misuse.