VaultGemma: A differentially private LLM (arxiv.org)

Google Research and DeepMind have announced the release of VaultGemma, a groundbreaking large language model (LLM) with 1 billion parameters that is fully trained using differential privacy (DP). This model marks a pivotal advancement in the development of privacy-preserving AI, as it mitigates the risk of unauthorized data leaks and prevents the memorization of sensitive information during the pretraining phase. By employing DP throughout the entire training process rather than just during fine-tuning, VaultGemma ensures that the model cannot inadvertently disclose personally identifiable information (PII) from its training corpus. For the AI/ML community, VaultGemma represents a significant step toward creating robust, privately trained models that maintain high utility while safeguarding user data. This release showcases a new methodology for applying DP that narrows the utility gap between private and non-private LLMs, advancing research in the field of safe AI. Key technical features of VaultGemma include a decoder-only transformer architecture, the use of global attention across layers, and advanced batch processing techniques like Truncated Poisson Subsampling. By making this model open-access, Google aims to lower barriers for researchers looking to innovate in privacy-preserving AI, ultimately empowering users and developers to build responsible, secure applications.