Distillation, Experimentation, and Integration of AI for Adversarial Use (cloud.google.com)

In late 2025, the Google Threat Intelligence Group (GTIG) reported a notable rise in the use of artificial intelligence (AI) by threat actors to enhance various phases of attacks, including reconnaissance, social engineering, and malware development. This update highlights "distillation attacks," a method where malicious actors extract proprietary knowledge from AI models, such as Google's Gemini, violating terms of service and facilitating intellectual property theft. These attacks are significant as they underscore the urgency for AI developers and organizations to bolster security measures, given the increasing sophistication of adversarial methods using AI tools. The report reveals that state-sponsored actors from countries like North Korea, Iran, China, and Russia are now leveraging large language models (LLMs) for diverse malicious activities, including generating nuanced phishing lures and conducting reconnaissance. By employing AI in these strategies, attackers can operate more rapidly and efficiently, creating personalized phishing communications that are harder to detect. This trend not only poses heightened risks for organizations using AI but also illustrates a shift in the threat landscape, where adversarial use of generative AI becomes a major focus area for security professionals. To counteract these threats, Google emphasizes its commitment to developing AI responsibly while implementing proactive measures to disable malicious operations and improve model defenses.