The Thinking Leakage in Reinforcement Learning for LLMs (arxiv.org)

A recent study has unveiled a phenomenon termed “thinking leakage” in hybrid reasoning models, particularly in the Qwen3 series, which utilize a dual operational mode—THINK and NO-THINK—during training. The research demonstrates that improvements in reasoning performance, typically attributed to reinforcement learning (RL) conducted in NO-THINK mode, are primarily due to the reactivation of pre-existing reasoning capabilities from the model's THINK mode rather than the development of new reasoning skills. This finding could reshape understanding of model training and performance evaluation in reinforcement learning contexts. This insight is significant for the AI/ML community as it highlights the importance of recognizing latent capabilities within models and the potential for training methods to inadvertently exploit these instead of fostering genuine advancements. The study involved an in-depth analysis of model weights, activations, and attention heads. Notably, scaling the RL updates led to behavior resembling THINK-mode outputs; however, it fell short of outperforming those original capabilities. By investigating the underlying mechanisms, the research revealed that responses encouraged by the advantage-weighted update during training tended to align the models closer to their THINK state, underscoring the nuanced interplay between RL techniques and inherent model features.