Do looped transformers write reasoning they don't use? (igort99.github.io)

Recent experiments on looped transformers, particularly examining the model Huginn, revealed insights into how such models process and utilize reasoning. Leverage was placed on the concept of looped transformers, which share weights across multiple iterations of the same transformer block, potentially increasing depth without a corresponding rise in parameters. While earlier research suggested that models like Huginn may not effectively utilize the reasoning they generate, the current investigation aimed to explore whether increased loops influenced their reasoning outputs. Tests conducted with Huginn and a traditional transformer (Qwen) demonstrated that while increasing the count of loops improved performance, the model's reliance on reasoning traces remained significant. The findings indicated that Qwen outperformed Huginn, especially in removing reasoning traces, which often rendered Huginn dependent on its written steps for fewer operations. Notably, when tasked with simple arithmetic problems, Huginn struggled to calculate effectively without generating a reasoning trace, while Qwen could often arrive at solutions without needing to write down the intermediate steps. The results emphasize that while looped transformers can enhance computational depth, they may not inherently bolster reasoning capacity, challenging assumptions about their efficacy in AI/ML contexts and highlighting the need for deeper interpretability in language models.