Transformer Can Hold Two Thoughts at Once: Evidence of Linear (arxiv.org)

Recent research has unveiled a fascinating feature of Large Language Models (LLMs), proposing that they can simultaneously handle two distinct thoughts through a phenomenon described as "Superposition Linearity." This study highlights that despite the inherently non-linear nature of LLMs, they display a fundamental linearity: when combining inputs from different text streams, the models generate a superposition of their next-token distributions. This finding underscores that superposition is an intrinsic characteristic of the Transformer architecture, rather than a mere byproduct of training, although it lessens as pretraining advances. The implications of this research are significant for the AI/ML community, suggesting that fine-tuning can effectively restore this linearity, thus improving model output coherence. Furthermore, the introduction of a guided decoding method allows for the disentanglement of these superposed outputs, enabling the generation of two coherent continuations in a single forward pass. This breakthrough not only enhances the versatility and efficiency of LLMs but also opens new avenues for exploring multi-threaded conversations and richer contextual understanding in AI systems.