Bye Bye Perspective API: Lessons for Measurement Infrastructure in NLP, CSS and LLM Evaluation (arxiv.org)

The Perspective API, a widely used tool for measuring toxicity in online content, is set to shut down at the end of 2026, highlighting the vulnerabilities of relying on external tools for crucial measurements in natural language processing (NLP) and machine learning (ML). Researchers have found that their dependence on this API has led to significant challenges, including inflated research claims, unpredictable shifts in results due to silent model retraining, and unexplainable disparities in measured data. The reliance on Perspective throughout the lifecycle of large language models (LLMs) revealed fundamental flaws—particularly that it often rewarded system errors rather than correcting them. In response to this impending closure, researchers are advocating for the development of a self-governed measurement infrastructure within the NLP community. They propose a framework that includes ten essential requirements for such an infrastructure, emphasizing that the primary barrier to its creation is not technical capability but rather the value assigned to infrastructure work by the research community. As a proactive step, the researchers have also released Perspective scores for 5.9 million text snippets across 77 datasets, providing a lasting resource for continued study even after the API's shutdown.