LLMs Privacy Is Not Just Memorization (arxiv.org)

This position paper argues that privacy risk in large language models is far broader than the well-studied problem of memorized training examples. The authors present a lifecycle taxonomy of threats—from data collection and labeling practices, to inference-time context leakage, to autonomous agent behaviors and “deep inference” attacks that combine models with auxiliary data to extract sensitive signals. Through case studies and a longitudinal review of 1,322 AI/ML privacy papers (2016–2025), they show research attention has been heavily skewed toward memorization while many practical, scalable harms (context leakage, reidentification via model outputs, surveillance-capable agent chains) are understudied and poorly mitigated by current technical tools. Technically, the paper highlights that defenses focused on training-time interventions (e.g., preventing verbatim memorization or applying differential privacy) leave open attack surfaces at deployment and orchestration layers: prompt contexts, chained agents that reason across sessions, and inference-time probes that infer attributes rather than exact data. The authors call for new benchmarks, threat models, evaluation methods, and interdisciplinary responses—combining systems design, legal frameworks, human-centered policy and socio-technical governance—to address these emergent risks and reorient research priorities beyond memorization.