Demystifying Hidden-State Recurrence (arxiv.org)

A new framework called SWITCH has been proposed to enhance latent reasoning in reinforcement learning (RL) by addressing challenges associated with hidden-state recurrence. Traditionally, optimizing hidden-state structures with standard on-policy RL has been difficult and causally hard to interpret. The SWITCH framework introduces two explicit boundary tokens, <swi> for entry and </swi> for exit, which facilitate both compatibility with RL and enable direct mechanistic analysis. This allows the model to effectively switch between latent and visible states while maintaining clear decision points for the RL policy. The significance of SWITCH lies in its ability to integrate rigorous reasoning and transparency into RL models. It outperforms previous hidden-state-recurrence methods and demonstrates that such latent reasoning can be both trainable through RL and amenable to causal analysis. Notably, the research reveals that the switching tokens do more than just mark boundaries; they indicate a localized decision-making process that enhances the reasoning capabilities of the model. This work not only improves how models can learn and reason in complex environments but also opens new avenues for analyzing and understanding the inner workings of RL systems.