Legibility Is Not Interpretability (arxiv.org)

A recent study titled "Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning" raises critical questions about how reasoning steps in large language models (LLMs) are evaluated. While chain-of-thought reasoning is perceived as providing clarity into model decision-making, this research reveals that the textual representation of reasoning steps does not reliably convey their functional importance. Utilizing Monte Carlo rollouts to assess the expected reward associated with each reasoning step, the study found that even advanced LLM judges can only partially identify the high-advantage steps that contribute to correct outcomes. Fine-tuning models as step-level critics improved performance for incorrect answers, yet there remains a considerable gap for correctly reasoned responses. This work holds significant implications for the AI/ML community, particularly in the context of model interpretability and process reward modeling. It cautions against equating legibility with true interpretability, emphasizing that the perceived clarity of reasoning traces may not equate to understanding their actual impact on decision-making. As AI continues to integrate into critical applications, ensuring that models are not only legible but also interpretable is essential for building trust and implementing effective supervision mechanisms.