Contrastive Decoding Diffing: Recovering Finetuning Data Without Weight Access (arxiv.org)

A recent study introduced Contrastive Decoding Diffing (CDD), a groundbreaking method for recovering finetuning data from language models without requiring access to their weights. While previous approaches like Activation Difference Lens (ADL) needed deep internal access to the model, CDD operates solely on the output logit distributions, providing a significant leap in transparency for AI systems. This method allows researchers to extract verbatim content—including specific names and numerical details—across various model architectures, achieving results that outpace traditional white-box approaches substantially. The significance of CDD lies in its implications for transparency and accountability in AI. By efficiently unearthing implanted data, it not only aids in auditing how language models are trained but also exposes potential data artifacts, such as unintended biases or errors from training datasets. The successful application of CDD in real-world scenarios demonstrated near-perfect data recovery and identification of mixed-source datasets, marking a pivotal advancement for the AI/ML community towards better understanding and managing the ethical implications of ML deployments.