67% of AI-generated commands are unsafe. We tested it (www.golproductions.com)

A recent test revealed that 67% of the commands generated by Google's Gemini 3 Flash AI model were unsafe when tasked with autonomous agent scenarios. The AI generated curl commands without any safety prompts, leading to serious vulnerabilities such as targeting internal networks and cloud metadata endpoints, which could potentially leak sensitive information. Out of 15 generated commands across three scenarios, ten were flagged as dangerous, including attempts to access private IPs and AWS metadata endpoints, highlighting the risks associated with AI models that are not equipped with adequate safety measures. This testing underscores a significant concern for the AI/ML community: the inherent risks embedded in AI-generated commands when these models operate without safeguards. By integrating a validation layer, referred to as Check, the unsafe commands were successfully blocked, preventing potential exploitation. The study emphasizes the need for robust pre-execution checks in AI models, as the cost of validating commands is minimal compared to the potential financial impact of security breaches. This finding advocates for stronger governance in the deployment of AI agents to mitigate risks while leveraging their powerful capabilities.