Evaluating Netflix Show Synopses with LLM-as-a-Judge (netflixtechblog.com)

Netflix has implemented an innovative approach to enhance the quality of show synopses using a Large Language Model (LLM) as a judge. This technique addresses the challenge of evaluating the hundreds of thousands of synopses present in its catalog, which is essential for ensuring that users are presented with accurate and engaging content. The LLM evaluates synopses based on two primary quality dimensions: Creative Quality and Member Implicit Feedback, achieving over 85% agreement with human writers. This automated system not only streamlines the scoring process but also correlates high-quality scores with improved viewer engagement metrics, allowing Netflix to identify potential issues before a show’s debut. The significance of this development lies in its potential to transform content presentation in streaming services. By integrating advanced AI techniques like Automatic Prompt Optimization and consensus scoring, Netflix can achieve a scalable solution that maintains high editorial standards while meeting diverse viewer preferences. The LLM employs tiered rationales and agents for factuality checks to ensure accuracy across various aspects of synopses. This sophisticated framework optimizes creative evaluations and provides valuable insights into member behavior, helping Netflix refine its content recommendations and ultimately boost viewer satisfaction and retention rates.