The world model remembers, the actor forgets: dissecting AI forgetting on 1 GPU (arxiv.org)

Researchers have presented new findings on catastrophic forgetting in model-based reinforcement learning (RL) agents, specifically within the DreamerV3 family. Their study reveals that while the world model effectively retains knowledge about previous tasks—showing near-complete retention of reward discrimination and value estimates—the actor component of the agent significantly forgets how to execute learned tasks. This insight indicates that forgetting is more about the interaction channels than a lack of memory in the world model. Notably, even when the world model’s knowledge is utilized, interventions show that reinforcement learning from imagination fails to recover lost skills without prior experience interaction. The implications of this research suggest a paradigm shift in understanding continual learning in RL systems. By introducing graded dream rehearsal, a method that leverages the learned experiences of the world model without altering the actor's parameters, the authors demonstrate improved retention rates across various task sequences—marking a significant advancement in creating robust continual learners. This technique not only outperformed traditional replay methods but also introduces a systematic approach to addressing forgetting in RL, paving the way for more resilient AI systems capable of tackling complex, multi-task environments efficiently.