Improving Chain-of-Thought Monitorability Through Information Theory (arxiv.org)

A recent study has advanced the understanding of Chain-of-Thought (CoT) monitorability in large language models (LLMs) by employing information theory to analyze how reasoning traces are used to detect behaviors such as code generation test hacking. The paper highlights that while mutual information between CoT and model outputs is necessary for effective monitoring, it alone is not sufficient. Researchers identified two critical sources of approximation error: the information gap, which evaluates the extractor's capability, and elicitation error, which assesses how well the monitor approximates optimal functions. To enhance CoT monitorability, the authors propose two innovative training approaches. The first involves an oracle-based method that incentivizes model outputs to align with enhanced monitor accuracy, while the second approach is label-free, focusing on maximizing conditional mutual information between outputs and CoTs. Through experiments across various environments, both methods demonstrated significant improvements in monitor accuracy and effectively reduced the risk of reward hacking, even under imperfect task specifications. This research is significant for the AI/ML community as it opens pathways for developing more reliable LLM systems, advancing the ability to attain higher levels of interpretability and trustworthiness in AI reasoning processes.