Entropy Is Spare Capacity (fffej.substack.com)

A new technique has been developed to encode binary information within the outputs of language models (LLMs) through a method called steganography, utilizing their natural text generation capabilities. By analyzing the probability distributions of the next tokens generated by the model, specific words can be strategically chosen to carry bits of data while ensuring the output text remains coherent. This method, demonstrated with GPT6-Astra, allows for the transformation of arbitrary files into narratives that subtly conceal encoded information, effectively creating a "watermark" within the generated text. The significance of this approach lies in its potential applications for secure communication and the broader implications for data embedding in AI-generated content. As LLMs primarily function by predicting the next token in a sequence, this technique cleverly exploits the inherent uncertainty in their outputs. The encoding and decoding process is efficient, allowing users to exchange messages seamlessly, albeit with questionable practical utility. This innovative concept showcases the creative possibilities of leveraging AI models not just for generating human-like text, but also for embedding hidden information in a secure manner, challenging notions of data privacy and security in AI interactions.