Ouroboros: Dynamic Weight Generation for Recursive Transformers (arxiv.org)

Researchers have introduced Ouroboros, a novel approach to enhance recursive transformers by implementing dynamic weight generation through input-conditioned LoRA modulation. This method addresses a significant limitation in conventional recursive transformers, which apply the same transformation at each depth step, hindering their ability to execute distinct operations. By integrating a compact Controller hypernetwork that generates a per-step modulation vector based on the hidden state, Ouroboros enables each recurrence step to be more responsive to input variations. This innovation is paired with gated recurrence and per-step normalization techniques for stability, ultimately resulting in a substantial 43.4% reduction in training loss compared to a traditional 17-layer configuration. This breakthrough is particularly important for the AI/ML community as it redefines the potential of recursive transformer architectures, allowing for more efficient and adaptable models with fewer parameters—only adding 9.2 million trainable parameters while outperforming static counterparts by 1.44 loss points across various depths. Although the Controller demonstrates outstanding results during training, its performance on unseen data remains to be optimized, indicating ongoing challenges in model generalization. The findings underscore the importance of gated recurrence and set the stage for advances in transformer adaptability and efficiency.