Automated red teaming with RL: attacker-defender co-training (castform.com)

A team has successfully implemented automated red teaming using reinforcement learning (RL) to enhance safety measures on the qwen3.5-4b model. By training an attacker with RL to exploit vulnerabilities across 366 harmbench behaviors, the researchers established a dynamic attacker-defender loop where both sides evolve through iterative training. This automation allows for a continuous improvement cycle, resulting in an impressive 92% defense rate against attacks while maintaining an 88% accuracy on benign tasks, despite a slight drop from the baseline. The significance of this approach lies in its departure from conventional supervised fine-tuning methods, which rely on known jailbreaks, by optimizing directly for compliance through RL. This method encourages the discovery of novel attack strategies that might not be represented in existing datasets. The use of diversity clustering rewards helped the attacker explore various tactics, preventing it from converging on a single approach. This demonstration not only showcases the potential for fully automated RL in red teaming but also sets the stage for further development in creating more robust AI models capable of discerning harmful content from benign interactions.