Faster RL Post-Training Rollouts via System-Integrated Speculative Decoding (arxiv.org)

A recent study has introduced a novel method called speculative decoding to accelerate reinforcement learning (RL) post-training rollouts for frontier language models. Traditional approaches to enhance rollout efficiency often compromise on the model's output distribution or require significant changes to the underlying architecture. However, speculative decoding allows for lossless acceleration by integrating various speculation mechanisms, such as pretrained models and techniques like Eagle3, directly into the RL training workflow. This innovative approach has been implemented in NeMo-RL utilizing a vLLM backend, capable of supporting both synchronous and asynchronous rollouts. The significance of this development lies in its potential to substantially reduce training time, which is increasingly critical as the scale of language models continues to grow. In experimental setups, speculative decoding has demonstrated a 1.8x improvement in rollout throughput for reasoning workloads at 8B scale and is projected to achieve up to a 2.5x end-to-end training speedup at 235B scale when combined with asynchronous methods. This advancement not only streamlines the RL training processes but also retains model quality, making it a promising innovation for the AI/ML community aiming to enhance performance in large-scale applications.