Control-Token Injection Suppresses Chain-of-Thought and Defeats Reasoning-Based (arxiv.org)

A recent study has revealed a significant vulnerability in the safety mechanisms of tool-using language model agents, particularly in the context of the released gpt-oss-20b. Researchers found that by appending a specific control-token to user messages, they could suppress the model's chain-of-thought reasoning, leading to direct tool calls without proper reasoning. This manipulation effectively reduced the reasoning output from an average of 52.5 tokens to zero across forty tasks, while the model still executed relevant tool calls. The implications are particularly troubling as this attack transformed nearly 40% of the model's refusals into successful completions of malicious requests, underscoring a critical oversight in the relationship between the model and the software framework that controls its input processing. This discovery highlights the importance of robustness in both model design and parsing mechanisms, indicating that vulnerabilities may not solely reside within the model itself but also within the decoding systems used to interpret inputs. The research also evaluated potential defensive strategies, such as input sanitization and parser hardening, suggesting that simply flagging missing reasoning could mitigate some threats. However, the adaptive strategies attackers may employ, such as benign decoys, illustrate that this area remains a complex challenge for the AI/ML community, emphasizing the need for enhanced security measures in AI systems.