The Art of Scaling Reinforcement Learning Compute for LLMs [Meta] (arxiv.org)

Researchers present the first large-scale, systematic study of how reinforcement learning (RL) training for large language models (LLMs) scales with compute, using over 400,000 GPU-hours to build a principled framework akin to pre-training scaling laws. They fit sigmoidal compute–performance curves for RL, run broad ablations across common design choices, and identify which elements change asymptotic performance versus those that mainly affect compute efficiency. The authors package these insights into a practical recipe, ScaleRL, and validate its predictive power by scaling a single RL run to 100,000 GPU-hours and accurately forecasting validation performance. Key technical takeaways: stable, well-chosen RL recipes follow predictable scaling trajectories that allow extrapolation from smaller jobs; not all algorithmic variants reach the same performance ceiling (asymptote); while choices like loss aggregation, normalization, curriculum design, and off-policy vs on-policy methods largely modulate efficiency rather than shift the asymptote. The work makes RL training for LLMs more measurable and predictable, enabling better evaluation of algorithmic improvements, more reliable resource planning for large-scale RL experiments, and a clearer path toward reproducible, cost-effective RL at scale.