We can't have nice things because of AI scrapers (blog.metabrainz.org)

The MetaBrainz team has recently taken decisive action against AI scrapers that have been disregarding standard web protocols, such as robots.txt, to unlawfully gather data for their models. By scraping MusicBrainz one page at a time, these companies have overwhelmed servers, disrupting access for legitimate users. In response, MetaBrainz has implemented several critical changes, including requiring an Authorization token for their /metadata/lookup API endpoints and removing certain ListenBrainz Labs API endpoints that were prone to abuse. This situation highlights significant concerns within the AI/ML community regarding ethical data sourcing and the respect for web infrastructure. As companies increasingly rely on vast datasets, the tension between data acquisition for AI development and the operational integrity of service providers is becoming more pronounced. MetaBrainz's measures not only safeguard their services but also serve as a cautionary tale for how AI scrapers may undermine the ecosystem, emphasizing the need for compliance and appropriate use of publicly available data.