In Defense of AI Evals, for Everyone (www.sh-reya.com)

The author pushes back against a rising “anti-evals” sentiment by defining evals as the systematic measurement of application quality—not a single metric or rigid protocol, but continuous, deliberate testing and error analysis. They argue that almost every successful AI product already does evals (even if informally): foundation models undergo pretraining and posttraining (supervised fine-tuning, RLHF, preference tuning), providers report task-specific scores (math, coding, instruction following, tool use) and participate in public benchmarks like LMArena, and product teams analyze private API traces to guide improvements. Because upstream eval work is extensive, some teams (e.g., building coding agents) may feel they can skip formal evaluation, but that’s often just leveraging others’ rigor. Technically, the post stresses when light-touch evals can suffice (tasks well represented in posttraining or teams with deep domain expertise and relentless dogfooding) versus when rigorous, decomposed evals are essential—complex document processing, long-context retrieval, or novel tasks where failure modes are subtle. Practical techniques include task decomposition, targeted error analysis, and scalable approaches like LLM-as-Judge. Dismissing evals is harmful for newcomers who lack experience or analytic processes; instead, the community should expand accessible evaluation techniques, teach them broadly, and treat evals as a spectrum adaptable to each project’s risks and needs.