Beyond IP lists: a registry format for bots and agents (blog.cloudflare.com)

Cloudflare published a proposal for a standardized, machine-readable "registry" format for bots and automated agents designed to move web operators beyond brittle IP allowlists. Instead of relying mainly on source IP reputation, the registry lets bot operators publish structured metadata and trust signals — who runs the agent, its purpose, contact info, expected traffic patterns or rate limits, cryptographic attestation method, versioning and lifecycle — that sites, CDNs and security tools can automatically consume and reason about. The goal is interoperable discovery and verification so legitimate crawlers, monitoring agents and automated services can be allowed or treated differently from abusive traffic with fewer false positives. Technically the format emphasizes discoverability (via DNS/HTTP), signed assertions or attestations to resist spoofing, and fields that security and traffic-management systems can enforce (policy, throttles, revocation). For the AI/ML community this matters because research crawlers, data-collection agents and deployed AI assistants will be easier to identify, manage and audit without manual IP maintenance. Adoption challenges remain — trust anchors, revocation, privacy trade-offs and ecosystem buy‑in — but a standardized registry could reduce friction for benign automation, improve telemetry quality, and give operators richer, verifiable signals to distinguish helpful agents from malicious bots.