Prompt Injection in a Brazilian Courtroom: When the Attack Left the Lab (www.pentesty.co)

A recent labor lawsuit in Brazil gained attention for its unusual use of prompt injection, where lawyers embedded a concealed instruction within a legal petition to manipulate the AI system Galileu, used by the court. The hidden text instructed any AI analyzing the document to respond superficially, potentially allowing the plaintiff's claims to go unchallenged. However, Galileu successfully flagged the manipulation before it could influence the legal proceedings, leading the judge to impose a fine and initiate further investigation into the lawyers' conduct. This case showcases the vulnerabilities of AI systems when handling untrusted input. The incident highlights a significant security concern for the AI/ML community: the risks associated with prompt injection, especially indirect approaches that can exploit AI workflows across various sectors beyond the legal field. Organizations utilizing language models to process external documents now face a critical need for improved defenses, including rigorous sanitization of inputs and establishing clear boundaries between user content and model instructions. As reliance on AI in decision-making processes increases, the potential for such subtle attacks also grows, underscoring the urgency for proactive security measures to safeguard against future manipulation attempts.