AI watermarking could make LLM guardrail adherence unpredictable — and that could be a big problem for the EU AI Act (www.techradar.com)

A recent study highlights that AI watermarking, designed to authenticate text as AI-generated or human-written, can inadvertently alter the behavior of large language models (LLMs), leading to significant security concerns. Researchers found that techniques like SynthID-Text can influence critical decisions made by models regarding harmful prompts and susceptibility to prompt injection, potentially increasing the likelihood of these models providing unsafe outputs. This phenomenon, termed "sampling drift," could undermine the effectiveness of safety measures LLMs are meant to uphold. The implications of this research are particularly pertinent to the upcoming EU AI Act, which may drive broader adoption of watermarking across AI models to ensure transparency and compliance. However, the unintended consequences highlighted by the study—where watermarking could compromise the models' refusal to engage with harmful requests—underscore the need for developers to rigorously test and reassess their systems whenever watermarking is integrated. As AI watermarking becomes mainstream, these findings prompt a reevaluation of its implementation and safety protocols, stressing the balance between authenticity and security in AI development.