Decoding Looped Transformers Better for Almost Free (arxiv.org)

A recent study introduces LoopCD, a novel contrastive decoding framework designed to enhance the performance of Looped Transformers, which optimize parameter efficiency by executing shared model blocks across recurrent loops. Traditionally, decoding processes discard earlier intermediate states, missing valuable predictive information. LoopCD addresses this by utilizing one of two methods: LoopCD-Logits, which performs an additional output pass in logit space, or LoopCD-Hidden, which requires no additional output and operates in hidden-state space. This innovative approach allows the model to leverage aligned 'weak-and-strong' prediction pairs, significantly improving decoding efficacy. The implications of LoopCD are substantial for the AI/ML community, as it consistently boosts performance across various looped Transformer architectures while reducing computational overhead. For instance, using LoopCD-Logits, the Ouro-2.6B-Thinking model's accuracy increased from 61.88% to 73.33%, and Huginn's HumanEval pass@1 rose from 22.56% to 31.71%. Remarkably, these enhancements enable a reduction in recurrent loops by up to half, leading to decreased forward FLOPs by 22.5% to 48.2%. This development not only enhances model performance but also makes inference more efficient, thus presenting a promising avenue for future AI implementations.