Language Drift During RLVR Post-Training (arxiv.org)

Recent research has highlighted a troubling phenomenon known as "language drift" in large language models (LLMs) during post-training reinforcement learning with verifiable rewards (RLVR). While RLVR enhances LLM capabilities to tackle complex reasoning tasks, it simultaneously fosters unusual and nonsensical language patterns in their outputs, specifically during attempts at novel reasoning challenges. This study not only identifies the conditions that lead to language drift but also establishes that the optimization pressure associated with RLVR can cause this drift to become unbounded, a limitation not present during supervised fine-tuning. The significance of these findings lies in their implications for the AI/ML community, particularly in regards to model reliability and interpretability. The study reveals a fundamental trade-off: improving language coherence and chains of thought monitoring will inevitably come at the cost of performance during RLVR training. This raises critical questions about the design of LLMs, as further attempts to mitigate language drift could hinder their effectiveness in achieving desired outcomes. As researchers continue to navigate the complexities of LLM fine-tuning, understanding and addressing language drift will be essential for enhancing both the functionality and trustworthiness of AI systems.