Stealing Reasoning Traces from Proprietary LLM APIs (arxiv.org)

Recent research has unveiled a significant architectural vulnerability in the way leading large language model (LLM) providers handle the reasoning traces of their models. These chains-of-thought are typically encrypted and returned to clients to protect intellectual property. However, the study reveals that these encrypted blocks are interchangeable across different sessions and models within the same ecosystem. By exploiting this weakness, attackers can develop a scalable decryption method, gaining access to sensitive proprietary reasoning from various models without directly breaching the more advanced systems. The implications of this vulnerability are profound for the AI/ML community. It not only bypasses safeguards against information extraction but also allows for large-scale data retrieval, as demonstrated by the researchers' recovery of personally identifiable information and credentials from publicly available session logs. Furthermore, attackers can execute invisible prompt injections, embedding malicious code within the encrypted blocks. To counter these threats, the researchers propose robust cryptographic and system-level mitigations aimed at safeguarding client-side reasoning, highlighting the urgent need for enhanced security measures in LLM deployment.