A critical look at self-distillation for reasoning (jasonkena.github.io)

In a recent paper set to be presented at NeurIPS 2026, researchers from the MIT-IBM Computing Research Lab and MIT CSAIL critically assess the concept of self-distillation in the context of reinforcement learning. The study explores the effectiveness of self-distillation—a technique where models use privileged information to enhance their learning—by analyzing two approaches: Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD). The authors reveal significant shortcomings in self-distillation, such as overfitting and non-generalizable gains. They argue that while self-distillation was expected to enable models to learn from their experiences in a sample-efficient manner, in practice, it often hinders performance on new or diverse tasks. The implications of this work are substantial for the AI/ML community. The findings suggest that rather than improving reasoning through self-teaching, as initially hoped, self-distillation methods may lead models away from optimal performance due to flawed gradient directions. This critical examination urges researchers to rethink and develop more effective reinforcement learning algorithms that genuinely utilize privileged information to enhance learning outcomes. The necessity for robust and reliable mechanisms in RL is emphasized, potentially reshaping future approaches in AI training methodologies.