Perturb Your Data: Paraphrase-Guided Training Data Watermarking (arxiv.org)

A new watermarking method for training data, called SPECTRA, has been introduced, enhancing the ability to track the origins of data used in the training of Large Language Models (LLMs). This technique is particularly crucial as LLMs frequently rely on extensive text datasets sourced from the internet, posing challenges for copyright enforcement and data licensing. SPECTRA utilizes an approach that involves paraphrasing training text while maintaining its original semantic score, thereby ensuring identification without altering the statistical distribution of the data. Notably, it can detect the watermark even when the data constitutes less than 0.001% of the training corpus. The significance of SPECTRA lies in its impressive detection capabilities, boasting a p-value gap exceeding nine orders of magnitude when differentiating between models trained on watermarked data versus those that are not. This substantial improvement over existing baseline methods empowers data owners with a scalable solution that can be implemented before the release of their datasets. With SPECTRA, the AI/ML community gains a robust tool to protect intellectual property rights and promote accountability in the utilization of training data, ensuring that ethical standards are upheld in model development.