Distillation Defenses Easily Break After Reinforcement Learning (arxiv.org)

Recent research has highlighted significant vulnerabilities in distillation defenses against attacks on large language models (LLMs). Distillation attacks allow adversaries to replicate the reasoning capabilities of state-of-the-art LLMs by collecting reasoning traces and training their own models. While existing defenses have been evaluated under the assumption that attackers do not engage in further training, this paper demonstrates that incorporating reinforcement learning after distillation can easily undermine these protections. As a result, attackers can effectively lower the threshold for a successful distillation attack, leveraging accessible API data to steal reasoning capabilities that rival more complex attacks. This discovery is crucial for the AI/ML community as it challenges the efficacy of current defense mechanisms against model theft, revealing that many purportedly robust defenses may be rendered ineffective under realistic conditions. The research suggests that any defense leaking sufficient information to reconstruct reasoning traces is at risk. The authors advocate for exploring batch-level distillation defenses as a potentially more effective deterrent against such attacks, emphasizing the need for a reevaluation of security strategies in the evolving landscape of AI models.