VaultGemma: The most capable differentially private LLM (research.google)

Google Research and DeepMind have unveiled VaultGemma, the most capable large language model (LLM) trained from scratch with differential privacy (DP). By integrating mathematically rigorous privacy protections directly into the training process, VaultGemma addresses a key challenge in AI: building powerful models without compromising sensitive data. The team developed new scaling laws that quantify the complex trade-offs between compute budget, privacy guarantee, and model utility when applying DP, revealing that larger batch sizes and carefully balanced noise levels are essential for effective DP training. VaultGemma, a 1-billion parameter model, embodies these insights and sets a new standard for private AI by achieving strong empirical utility while rigorously bounding memorization of training data. Technically, VaultGemma was trained using innovative techniques like Poisson sampling combined with scalable DP-SGD algorithms to maintain fixed batch sizes while preserving strict privacy guarantees quantified as (ε ≤ 2.0, δ ≤ 1.1e-10) at the sequence level. The model’s final training loss closely matched predictions from the scaling laws, validating the team’s theoretical framework. Performance benchmarks demonstrate VaultGemma’s utility matches that of comparable non-private models from roughly five years ago—a significant milestone signaling that the privacy-utility gap can be systematically reduced. VaultGemma’s open release of model weights and documentation on Hugging Face and Kaggle invites the AI community to build on these advancements, promoting responsible, privacy-respecting AI development. This work marks a pivotal step in reconciling differential privacy with large-scale language modeling, empowering researchers to develop next-generation AI systems that are both powerful and privacy-preserving by design.