Deepfake Detector Robustness Testing (huggingface.co)

A new evaluation benchmark, the Social Media Robustness Benchmark, has been released to assess the robustness of deepfake detection models under real-world conditions. Unlike traditional test sets, which often fail to predict performance in actual social media environments, this benchmark features 40,574 image rows, including 2,400 clean images (1,200 real and 1,200 generated) and variations subjected to common platform-specific perturbations like JPEG compression and resizing across major platforms such as Instagram, Facebook, and TikTok. This resource is significant for the AI/ML community as it provides a structured method for measuring detection model performance under demographically balanced conditions and varied media processing pipelines. By utilizing paired configurations, the benchmark allows researchers to compute detailed performance metrics like paired AUC deltas, enabling more accurate comparisons of different detector architectures. As the dataset is intended solely for research evaluation and not for training, it emphasizes a responsible approach to advancing deepfake detection capabilities, highlighting the necessity of adapting detection models to handle real-world applications effectively.