Show HN: Livenerf – a benchmark for whether Opus 5.5 gets nerfed (github.com)

A new project called Livenerf has been launched as a benchmarking tool to determine if AI models, specifically Claude Opus 5.5, degrade in performance after their release. Livenerf addresses ongoing concerns within the AI community about potential "nerfs"—unannounced downgrades to model performance—by establishing a deterministic methodology to track changes. The benchmark captures performance on a “frozen panel” of tasks that are sampled consistently over time, aiming to provide a clear baseline for model evaluation right from launch. Significantly, Livenerf employs a stringent statistical analysis framework based on Anthropic's own guidelines, employing a series of exact grading and evaluation procedures. It ensures that any potential changes in performance are quantifiable and public, reporting both improvements and regressions with transparency. With the aim of creating a standardized measure of model consistency, Livenerf is positioned to act as a critical resource in the ongoing discourse surrounding model reliability and integrity within the AI/ML field. This could foster greater accountability among model developers and enhance trust within the AI user community.