Diversity as the Bottleneck in Self-Play (ivison.id.au)

A recent exploration into self-play training for AI models has revealed that diversity is a critical bottleneck for achieving progress toward artificial general intelligence (AGI). The study, which revisited the Absolute Zero framework, highlighted how models fail to maintain diversity in generated tasks when they are trained in a self-play loop. Initially, the model produced varied programming tasks, but as training progressed, it began to generate increasingly similar outputs. This issue arose partly due to a bug related to how the model's task generation buffer was managed, but even after the bug was fixed, diversity issues persisted, indicating a fundamental challenge in self-play design. The findings emphasize that while self-play can be an exciting avenue for model development, relying solely on reinforcement learning without effective diversity metrics and rewards can lead to performance collapse. Multiple strategies were tested to introduce diversity, such as embedding-based rewards and conditioning on diverse datasets, but they all eventually succumbed to similar output patterns. This research underscores the need for innovative approaches to maintaining diversity in AI training paradigms, as achieving robust performance may depend on overcoming these challenges to avoid convergence on repetitive solutions.