Out-of-Distribution Generalization in Transformers via Latent Space Reasoning (arxiv.org)

The paper tackles out-of-distribution (OOD) and compositional generalization in Transformers by equipping them with mechanisms for iterative, latent-space reasoning. Using a GSM8K-style modular-arithmetic on computational-graphs benchmark as a rigorous testbed, the authors propose a combined architectural recipe—input-adaptive recurrence, algorithmic supervision, a discrete bottleneck for anchored latent representations, and an explicit error-correction loop—that substantially improves generalization beyond the training distribution. This is significant because it targets a core limitation of current LMs: failure to systematically extrapolate algorithmic, multi-step solutions that require compositionality and recursion. Technically, input-adaptive recurrence lets the model perform conditioned iterative updates in latent space rather than relying solely on feedforward attention; algorithmic supervision provides loss signals on intermediate states to shape procedural computation; the discrete bottleneck anchors representations to a small set of stable latent codes, improving robustness and reuse; and the error-correction module detects and repairs mistakes across iterations. Empirically the combination yields scalable, native latent-space reasoning in Transformers and strong algorithmic generalization; mechanistic interpretability analyses show how these components produce modular, reusable operations in the model’s internal dynamics. The approach points toward architectural paths for more reliable, interpretable multi-step reasoning in future LLMs.