Self-Distillation Enables Continual Learning (self-distillation.github.io)

A groundbreaking advancement in continual learning has emerged with the introduction of On-Policy Self-Distillation Fine-Tuning (SDFT) by researchers from MIT and ETH Zurich. This innovative method enables models to learn new tasks without degrading previous skills, addressing fundamental challenges in the AI/ML community. By leveraging on-policy learning directly from expert demonstrations, SDFT generates training signals that help models retain their prior capabilities while acquiring new ones. In empirical tests, SDFT has demonstrated substantial improvements over traditional supervised fine-tuning (SFT), achieving higher accuracy in skill mastery and notably decreasing issues related to catastrophic forgetting. The significance of SDFT lies not only in its effectiveness but also in its ability to enhance in-distribution generalization. Unlike SFT, which relies on off-policy training that can lead to error compounding, SDFT trains models using their generated trajectories, making them more adept at recovering from mistakes. This property is amplified in larger models; SDFT outperforms SFT more dramatically as model size increases, suggesting that enhanced in-context learning leads to greater efficacy in continual learning. As AI models scale, the potential applications of SDFT could revolutionize how systems learn and adapt over time, paving the way for smoother integration of new skills without sacrificing previously acquired knowledge.