Evaluating Chain-of-Thought Monitorability (openai.com)

Researchers from OpenAI have introduced a systematic framework for evaluating the "chain-of-thought monitorability" of advanced AI reasoning models, such as GPT-5 Thinking. This approach focuses on understanding how these models make decisions by analyzing their internal thought processes rather than solely monitoring their outputs or actions. The significance of this research lies in its potential to enhance oversight of AI systems, particularly as they are deployed in high-stakes environments. The study finds that most modern reasoning models are relatively monitorable, with an effective correlation between longer reasoning efforts and improved monitorability. However, the challenge remains that changes in model training, data sources, and scaling can impact this monitorability. The evaluation framework consists of 13 tests categorized into intervention, process, and outcome-property evaluations, allowing researchers to measure how effectively a monitor can predict aspects of an agent's behavior. Notably, a trade-off was identified between reasoning effort and model size, suggesting that a smaller model performing at higher reasoning effort could achieve similar capabilities with better monitorability. The research highlights a "monitorability tax," where organizations may need to invest additional computational resources to ensure reliable oversight of AI systems as they scale, underscoring the need for robust monitoring strategies in the evolving landscape of AI development.