Generalized On-Policy Distillation with Reward Extrapolation (arxiv.org)

Researchers have introduced a new framework called Generalized On-Policy Distillation (G-OPD), which enhances the existing on-policy distillation (OPD) method used in machine learning. While OPD has already shown effective performance improvements in student models, G-OPD introduces a flexible reference model and a reward scaling factor to better balance reward and KL regularization in training. Significantly, the authors reveal that by increasing the reward scaling factor—termed reward extrapolation (ExOPD)—students can outperform their teachers, particularly when integrating knowledge from multiple domain experts. This development is crucial for the AI/ML community as it provides a robust theoretical foundation for OPD while demonstrating practical benefits through experiments involving math reasoning and code generation tasks. The findings suggest that employing a more accurate reward signal by leveraging a teacher’s pre-RL version can boost performance even further, although this method does require additional computational resources. Overall, G-OPD paves the way for innovative enhancements in model distillation techniques and encourages further exploration in this dynamic research area.