Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check (arxiv.org)

A new meta-analysis argues that “downstream scaling laws” — the practice of predicting how task performance will improve at larger model or compute scales from pretraining loss measured at smaller scales — are often unreliable. Surveying existing published results, the authors find that a close linear relationship between pretraining loss and downstream task performance holds in only 39% of cases. Moreover, apparently minor changes to experimental setup (task formulation, dataset, evaluation metric, or training regimen) can flip the observed scaling trend, and well-known phenomena such as emergence and inverse scaling further break simple predictive rules. For practitioners and researchers this is a cautionary result: blind extrapolation from pretraining loss can mislead budgeting, architecture choice, and claims about future capabilities. Technically, the paper implies we must move beyond single-parameter linear models and instead model conditional, task-specific relationships that account for nonlinearity, thresholds (emergence), and negative correlations (inverse scaling). The takeaway is to treat downstream scaling predictions as hypothesis-driven and uncertain, to report uncertainty and context, and to invest in richer empirical and theoretical models that explain when and why scaling laws do — or do not — generalize across tasks.