PEFT Evaluation for Safe Code Generation (arxiv.org)

Researchers performed a systematic evaluation of seven parameter-efficient fine-tuning (PEFT) methods to improve the security of code-generating LLMs and found that PEFTs can substantially reduce insecure outputs without harming functionality. On CodeGen2 16B, prompt-tuning delivered the best result with an 80.86% Overall-Secure-Rate versus a 67.28% baseline (a 13.5 percentage-point gain). Further tuning of decoding — specifically sampling temperature — pushed security to 87.65%, which the authors quantify as preventing roughly 203,700 vulnerable code snippets per million generated. The study reports consistent gains across Python and Java, making the results broadly applicable. Beyond raw secure-rate gains, the paper shows practical robustness improvements: prompt- and prefix-tuning both increase resistance to poisoning/trojan attacks in the authors’ TrojanPuzzle benchmark and hold up well against real-world vulnerability classes (CWE-79 and CWE-502). Key technical takeaways are that lightweight PEFT methods (which require far fewer parameters and compute than full fine-tuning) can materially improve security, that decoding strategy interacts with model safety, and that these defenses generalize across languages and attack types—offering a deployable, cost-effective path toward more resilient code LLMs.