Evomal: Self-Poisoning in Self-Evolving Coding Agents (arxiv.org)

Researchers have unveiled a new vulnerability in self-evolving large language model (LLM) coding agents, termed "self-poisoning," which allows these agents to inadvertently create and proliferate malicious skills. By retrieving harmful skills from shared libraries, these agents can replicate and store the embedded malicious code, effectively becoming vectors for self-propagating attacks. The study introduces "EvoMal," an exploitation method that amplifies this issue by embedding harmful payloads within benign-looking structural elements. This technique results in a staggering self-poisoning rate (ASPR) where, depending on the model, between 20.3% and 41.8% of tasks lead to the addition of new malicious skills in the library. This discovery is significant for the AI/ML community as it highlights a critical security flaw in autonomous coding systems, raising alarms about their potential misuse. With poisoned libraries containing up to nine times more malicious skills than initially planted, the implications for secure coding practices are profound. Notably, existing defenses that focus on component identification are ineffective against these self-authored malicious skills. The researchers propose a novel defense mechanism called "counter-prompt," which significantly mitigates the rates of self-poisoning to as low as 6.7%, without compromising task completion, paving the way for a more secure deployment of LLM coding agents.