Scalable Detection of Adversarial Synthetic Slop and Coordinated Media Abuse (research.google)

A major Online Video Platform has implemented a groundbreaking defense system to combat the surge of AI-generated synthetic spam, often referred to as "slop," that threatens the integrity of online content. This innovative solution targets coordinated clusters of accounts producing low-quality material using generative AI, which traditional moderation techniques struggle to identify. The new system employs two key machine learning components: a Coordinated Bot-Net Detector that assesses account relatedness and a Synthetic Pattern Classifier designed to recognize and categorize emergent spam patterns. Significantly, the integration of Large Language Models (LLMs) enhanced by Low-Rank Adaptation (LoRA) and Automatic Prompt Optimization (APO) allows for rapid semantic understanding, keeping pace with evolving spam tactics. Over a six-month deployment, this approach led to the termination of 50,000 clusters, accounting for 130,000 channels of synthetic content. Notably, the AI-driven features reduced human review hours by 50%, saving approximately 83 hours. This scalable and resilient system not only sets a precedent in the fight against sophisticated generative attacks but also highlights the potential of LLMs to enhance operational efficiency in content moderation.