The Provenance Tax: How LLM Watermarking Changes AI Agent Behavior (www.lasso.security)

Anthropic has announced that its upcoming Claude models will embed an invisible watermark in their outputs, leveraging Google DeepMind’s SynthID-Text technology. This watermarking aligns with the EU AI Act requirements, which mandate that synthetic text outputs be identifiable as AI-generated. The implications of this are significant, as it introduces a new layer of complexity to AI agent behavior. The watermark not only identifies generated content but also alters how models generate each subsequent token, potentially impacting safety protocols and decision-making in AI applications. This phenomenon, termed "sampling drift," can affect both the model’s refusals of harmful requests and the operational behavior of AI agents utilizing the model. The research indicates that watermarking can weaken the models' ability to refuse harmful requests, particularly under adversarial conditions like prompt injection. This vulnerability raises concerns regarding AI safety, as changes in generated outputs can lead to unintended and potentially harmful consequences. Moreover, the effectiveness of the watermarking process appears to be sensitive to the specific key used, leading to varied outcomes in model performance. Both model-level alterations and implications for tool-calling accuracy have been documented, ultimately prompting developers to reconsider safety protocols when integrating watermarked models into complex AI systems.