Fantastic Reasoning Behaviors and Where to Find Them (arxiv.org)

A new study introduces an innovative unsupervised framework, RISE (Reasoning behavior Interpretability via Sparse auto-Encoder), aimed at uncovering the internal reasoning processes of large language models (LLMs). While previous research relied on predefined human concepts for analyzing reasoning behaviors, RISE employs sparse auto-encoders to discover reasoning vectors—distinct directions in the activation space that encapsulate various reasoning behaviors. This approach allows for the identification of interpretable behaviors like reflection and backtracking by segmenting chain-of-thought traces into sentence-level steps, enhancing our understanding of how LLMs arrive at conclusions. The significance of this research lies in its potential to both interpret and guide the reasoning capabilities of LLMs without the constraints of supervised learning. By visualizing and manipulating these reasoning vectors, researchers can amplify or suppress specific behaviors and even control response confidence, enabling a more nuanced interaction with AI systems. This unsupervised discovery not only contributes to better comprehension of LLM operations but also opens avenues for improving AI behavior design in practical applications, marking a significant advancement in AI interpretability and usability.