A good teacher meets the student where they are (www.perplexity.ai)

Researchers have introduced a novel approach to reinforcement learning called Joint On-policy Learning and Teaching (JOLT), which addresses the challenges of sparse reward signals in language model training. In traditional reinforcement learning, models learn from outcome rewards, often leading to inefficiencies when results are rare or complex. JOLT enhances this by utilizing a single model as both a teacher, providing dense feedback, and a student, learning from it. Through joint training, the teacher tailors its guidance based on the student’s current capabilities, thereby improving the learning process across various complex tasks such as mathematical reasoning and code generation. The significance of JOLT lies in its ability to improve both model performance and training efficiency, as demonstrated in experiments where it outperformed existing methods like On-Policy Distillation (OPD) and Group Relative Policy Optimization (GRPO). JOLT and its enhanced version, JOLT+, showed substantial accuracy improvements—up to 4.2 percentage points higher in mathematical tasks—and demonstrated superior sample efficiency, processing training data more effectively than competitors, which could reduce costs significantly. This breakthrough not only suggests a paradigm shift in how reinforcement learning models can be trained for complex tasks but also opens new avenues for developing AI systems that better understand and adjust to learners' needs.