We ran OWASP attacks on 8 LLMs. Optimized small models beat frontier defaults (www.megacode.ai)

A recent study benchmarked eight large language models (LLMs) from Anthropic, Google, OpenAI, and xAI against a series of OWASP security attacks, including prompt injection and PII disclosure. The surprising outcome revealed that optimized smaller models outperformed their frontier counterparts, demonstrating that effective prompt optimization can enhance security significantly. The research showed that three of the four optimized small models achieved higher defense success rates (DSR) than all the tested frontier models running in their default configurations. The study's methods involved using system-prompt archetypes, known as SOULs, to standardize testing across diverse LLMs. Each model was rigorously tested against a set of 400 adversarial probes and then underwent optimization to improve their security postures. The key finding indicates that even with identical system prompts, varying models produced vastly different security outcomes. Post-optimization, 23 of the 24 configurations reached an impressive DSR of 0.94 or higher, signifying that model choice became nearly irrelevant when prompt optimization was applied. This suggests that the AI/ML community should prioritize security-focused prompt engineering alongside model selection to ensure robust defenses against adversarial attacks.