Fooling around with encrypted reasoning blobs (blog.cryptographyengineering.com)

A recent exploration into AI encrypted reasoning has unveiled intriguing aspects of how large language models (LLMs) like Claude and OpenAI's models handle internal reasoning states. The author, engaging in a personal project, discovered that while LLM APIs typically return obscured reasoning data, they also send an encrypted copy of the model's reasoning as part of their interaction protocols. This data is crucial for maintaining continuity in stateless API conversations, but it raises questions about security and the integrity of conversations, especially when it can be replayed across different sessions and accounts. The significance of this discovery lies in the implications of these reasoning "blobs." Although they appear secure, the ability to replay unmodified encrypted reasoning blocks may suggest a vulnerability, where a malicious actor could potentially manipulate inputs via JSON injections. This highlights the need for robust sanitation of chat inputs and awareness of how reasoning data can be unexpectedly used or misused. The research pushes the boundaries of understanding LLMs while cautioning about the need for ongoing vigilance in protecting sensitive data processed by these advanced models.