The dead weight loss of strictly isotonic regression (www.gojiberries.io)

Strict isotonic regression — commonly used to post-hoc calibrate model scores — enforces a nondecreasing stepwise mapping (via PAVA) that treats any local inversion on a small calibration set as miscalibration and merges neighboring scores into flat steps. With limited calibration data this hard shape constraint frequently pools distinct scores (e.g., 0.712 vs 0.718): a simple calculation shows adjacent slices differing by only δ=0.006 with ~100 examples each have roughly a 46% chance of a sign flip in empirical rates, so many genuine distinctions collapse into a few dozen tied values. The result can improve reliability but materially degrades resolution and downstream ranking/threshold decisions (ties replace ordered pairs; isotonic fits often coincide with the convex hull of the empirical ROC), so discrimination and decision utility can suffer even if aggregate calibration metrics look better. The important implication is that the problem is estimator shape and finite-sample behavior, not the scoring rule. The paper provides practical diagnostics to separate desirable noise-based flattening from undesirable limited-data flattening: bootstrap stability of steps, conditional AUC among tied pairs on holdouts, sweeping calibration sample size, a minimum-detectable-difference (MDD) calculation, slope tests via shape-constrained splines, and threshold-cost comparisons. It recommends softer monotone alternatives (nearly-isotonic regression, relaxed PAVA, smoothed/regularized isotonic, monotone splines) and parametric baselines (Platt/temperature scaling for tiny splits). The Calibre toolkit implements these monotone calibrators and resolution-aware diagnostics so practitioners can preserve as much useful ranking granularity as the calibration data legitimately supports.