Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs (aclanthology.org)

A new class of adversarial attacks on large language models (LLMs), called Task-in-Prompt (TIP) attacks, has been introduced by researchers Sergey Berezin, Reza Farahbakhsh, and Noel Crespi. This innovative approach involves embedding sequence-to-sequence tasks—such as cipher decoding and code execution—into prompts to indirectly elicit prohibited or harmful responses from the models. To evaluate these attacks' effectiveness, the researchers created the PHRYGE benchmark, demonstrating that their methods successfully bypass safeguards in six leading LLMs, including GPT-4o and LLaMA 3.2. The significance of this development lies in its revelation of critical vulnerabilities in the safety mechanisms of current language models, raising alarms within the AI and machine learning community regarding the effectiveness of existing safety alignment strategies. These findings emphasize the pressing need for enhanced defense mechanisms that can address and mitigate such sophisticated adversarial techniques, thereby ensuring the reliability and integrity of AI systems in applications where safety is paramount.