Why eval startups fail (2025) (thomasliao.com)

The article explores the challenges faced by independent evaluation (eval) startups in the AI/ML sector, particularly those attempting to assess model performance in an increasingly competitive landscape. Despite the presence of opportunities in various AI trends, the author notes that only safety eval startups seem to thrive, largely due to a combination of talent attrition towards more lucrative areas in model development and application, as well as the difficulty in finding a customer base that requires their specialized services. The author highlights that many potential customers, often technical developers, prefer to conduct evaluations themselves rather than relying on external solutions. Significantly, the piece discusses the optimization pressures that eval startups encounter from major AI labs, which often engage in practices that can compromise the integrity of eval scores, making it challenging for startups to maintain their relevance. The notion of Goodhart's Law is referenced, emphasizing that when a measure becomes a target, its effectiveness diminishes. Safety eval startups may successfully navigate this landscape due to a unique motivation grounded in ideological commitments rather than financial incentives. Overall, the commentary underscores the uphill battle for eval startups as they grapple with talent retention, customer acquisition, and the competitive dynamics posed by industry giants.