Good Teachers Don't Cheat (jasonkena.github.io)

Recent research has highlighted the shared objective functions among policy gradient reinforcement learning (RL), self-distillation techniques, and Pedagogical RL, specifically the optimization of \(\mathbb{E}_\pi[R] - \beta\KL{\pi}{\pi_0}\). These approaches utilize privileged information \(z\) to simplify the optimization process, but the key takeaway is that at optimality, this privileged information should not influence the outcome—emphasizing the notion that "good teachers don’t cheat." This principle raises important considerations about the validity of teacher-student training dynamics in AI, suggesting that reliance on direct expert guidance could hinder long-term learning effectiveness. The study underscores the ease of optimizing KL divergence compared to sparse rewards, paving the way for a two-stage process: firstly, training a teacher model \(g\) to meet the objective and subsequently distilling its knowledge into a student policy \(\pi\) without the privileged information. This process can reveal that existing on-policy distillation methods may inadvertently allow learners to depend on prior expertise, potentially leading to instability during training. The findings suggest that a more robust and efficient learning approach could emerge from refining these methods, thereby informing future advancements in AI education frameworks across the AI/ML community.