Privacy Leakage from Gradients in Split-LLM Training (arxiv.org)

A recent study has unveiled significant privacy vulnerabilities in split-LLM (Large Language Model) training systems. Researchers discovered that the gradient returned from an Untrusted Cloud Node (UCN) exposes real data through its structure, specifically revealing rows of real data interlaced with decoys. This flaw was identified despite passing initial privacy assessments, raising concerns about the effectiveness of current privacy measures in split-LLM setups. The team demonstrated that, across multiple experiments, their analysis could accurately determine which rows contained real data by detecting patterns of zeros in the returned gradients, indicating a potential risk for sensitive information leakage. The implications of this research are profound for the AI/ML community, as it highlights gaps in the security of distributed training frameworks commonly used to protect data privacy. Although the researchers proposed techniques like gradient clipping and noising to mitigate the leakage, they acknowledged that various attack methods remain untested and could still compromise the integrity of these systems. This study underscores the need for comprehensive evaluations of privacy mechanisms in AI training processes, especially as the deployment of split-LLMs becomes more widespread in sensitive applications.