Finetuning Activates Verbatim Recall of Copyrighted Books in LLMs (github.com)

Recent research reveals that finetuning large language models (LLMs) can lead to verbatim recall of copyrighted texts, particularly books, raising significant legal and ethical concerns. The study introduces a preprocessing pipeline and scripts that allow models to memorize and generate excerpts from texts, exemplified by Cormac McCarthy's *The Road*. This poses a challenge for the AI/ML community regarding how intellectual property is managed in the context of machine learning and model training. The researchers have developed various methodologies for finetuning, including the use of APIs from OpenAI, Google's Vertex AI, and Tinker, to demonstrate how easily LLMs can reproduce precise text from the training data. They provide extensive evaluation metrics to gauge the degree of memorization across different models and emphasize the need for robust copyright protections for literary works. As models become more capable of recalling specific copyrighted content verbatim, this study highlights the pressing need for clear guidelines and frameworks to address the potential consequences of using copyrighted texts in LLM training.