Beyond 80/20: High-Entropy Minority Tokens Drive Effective RL for LLM Reasoning (arxiv.org)

A recent study has explored a novel approach to Reinforcement Learning with Verifiable Rewards (RLVR), focusing on the role of high-entropy minority tokens in enhancing the reasoning capabilities of Large Language Models (LLMs). By analyzing token entropy patterns, researchers discovered that a small proportion of tokens with high entropy significantly directs the model towards effective reasoning pathways. This challenges the traditional 80/20 rule in machine learning, demonstrating that utilizing just 20% of these high-entropy tokens can yield performance comparable to full-gradient updates, while performing even better on larger models. This breakthrough holds significant implications for the AI/ML community, as it emphasizes the importance of optimizing specific 'forking tokens' rather than overwhelming the training process with low-entropy data. By refining RLVR training to focus on these critical tokens, researchers not only advance our understanding of the underlying mechanisms of RLVR but also pave the way for more efficient model training strategies. The findings suggest a strong scaling trend that could lead to more effective and resource-efficient LLMs, ultimately enhancing their reasoning capabilities and application in complex tasks.