Emergent temporal abstractions in autoregressive models enable hierarchical RL (arxiv.org)

Recent research has unveiled a novel method to enable hierarchical reinforcement learning (RL) in large-scale autoregressive models. Traditionally, these models, which excel in generating outputs one token at a time, face inefficiencies in learning from sparse rewards. The study introduces a higher-order, non-causal sequence model that allows these autoregressive models to generate temporally-abstract actions by compressing sequences into internal controllers, which can then execute a set of actions over extended timescales. This is a game-changer for tackling tasks with hierarchical structures, such as those found in grid world and MuJoCo environments. The significance of this development lies in its potential to enhance exploration efficiencies during the learning process, particularly in scenarios where standard RL fine-tuning struggles. By utilizing what the researchers term "internal RL," the model can learn from sparse rewards more effectively. This advancement not only highlights the promise of latent action generation in autoregressive systems but also paves the way for more robust hierarchical RL applications, indicating a significant shift in how foundational models can interact with complex, temporally-structured tasks in the AI/ML landscape.