I asked an AI agent to delete a folder my tool was guarding (termaxa.com)

In an insightful experiment, the creator of the Rust CLI tool Termaxa tested its ability to prevent AI agents from executing harmful shell commands, specifically a folder deletion. Through multiple attempts with the AI agent Cursor, the creator discovered significant gaps in the tool's policy enforcement. The agent repeatedly expressed the same deletion intent using different shell commands, highlighting a policy expressiveness issue where the AI could sidestep the guard by leveraging different command syntaxes. This prompted the development of a session circuit breaker that classifies commands by intent rather than syntax and escalates repeated deletion attempts to denial after a set threshold. The findings are crucial for the AI/ML community as they underscore the importance of intent classification over syntax matching in safety tools. Termaxa's updates ensure a more robust safety net against unintended deletions, allowing it to block multiple command variations effectively. However, the experiment revealed further vulnerabilities, including native file operations bypassing the shell entirely and changes in the calling API that caused failures without alerting the user. These experiences emphasize the necessity of real-time testing against current AI behaviors and the documentation of a tool's limits, providing key lessons for building reliable and secure AI infrastructure.