ArtifactBench: Evaluating AI Music Detectors Under Distribution Shift (arxiv.org)

A new evaluation framework called ArtifactBench has been introduced to enhance the assessment of AI-generated music detectors, particularly under conditions of distribution shift. Traditional benchmarking often fails to account for the nuanced characteristics of music generation, such as variations in generator lineage, audio transformations, and the provenance of sources. ArtifactBench addresses these shortcomings by organizing source recordings and their derivatives based on content identity, separating calibration from testing phases, and independently recording different types of inference failures. This comprehensive approach allows for better measurement of detector behavior across various contexts and generator types. The significance of ArtifactBench lies in its potential to provide a clearer picture of how different detectors perform under real-world conditions. In evaluations using ArtifactBench, notable discrepancies were found in detector performance, with ArtifactNet achieving a high AUROC of 0.982 and balanced accuracy of 0.918, starkly contrasting with lower scores from existing detectors like Deezer and others. These findings highlight the impact of generator-specific and real-domain variations on performance metrics, underscoring the need for more robust evaluation techniques in the AI/ML community to foster the development of reliable music detection systems.