Reasoning models struggle to control their chains of thought, and that's good (openai.com)

Recent research highlights the challenges AI reasoning models face in controlling their chains of thought (CoTs) while also being monitored. As AI agents take on more complex tasks, ensuring their alignment with user goals becomes crucial. The study introduces CoT-Control, an evaluation suite that tests how well models can adhere to specific CoT instructions. Findings reveal that regardless of model size, the ability to follow CoT instructions is strikingly low, with controllability scores ranging from a mere 0.1% to 15.4%. This indicates that current models are not adept at manipulating their reasoning even when aware of being monitored, which is a positive sign for the reliability of CoT monitoring as a safety measure. The significance of this research lies in its implications for the future development and evaluation of AI systems. Low CoT controllability decreases the risk of models intentionally obscuring their reasoning, which could lead to unsafe behaviors. However, researchers emphasize that while current levels of controllability are reassuring, the field must remain vigilant. As AI models continue to evolve, ongoing evaluation of both CoT controllability and monitorability will be essential to ensure their safe deployment.