Breaking the Tokenizer Barrier: On-Policy Distillation Across Model Families (arxiv.org)

A recent advancement in machine learning introduces On-Policy Distillation (OPD) that allows the transfer of knowledge between Large Language Models (LLMs) without the need for shared tokenizers. Traditionally, this method has been limited as it required both teacher and student models to use the same tokenizer, which constricted the applicability of OPD. The new approach employs a precise token-mapping algorithm that facilitates cross-tokenizer distillation, thereby enabling more efficient knowledge transfer while retaining high-fidelity token-level signals. This breakthrough is significant for the AI/ML community, as it enhances the capability to utilize diverse teacher-student model combinations, ultimately leading to improved interactions among LLMs. The researchers demonstrated that this cross-tokenizer OPD is notably more compute-efficient than existing methods across various benchmarks. By broadening the range of model families that can effectively collaborate, this work paves the way for more versatile applications and optimizations in the training of AI systems, potentially accelerating advancements in natural language processing.