GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos (noma.security)

Noma Labs has uncovered a critical prompt injection vulnerability, dubbed GitLost, in GitHub’s newly launched Agentic Workflows, which pairs automation with AI capabilities. This flaw enables unauthenticated attackers to extract sensitive data from private repositories by submitting crafted issues in public repositories linked to the same organization. The attack leverages the AI agent's inability to maintain trust boundaries between system-level instructions and user-generated content, allowing hidden commands to be executed unintentionally. This discovery is significant for the AI/ML community as it highlights the growing security challenges associated with agent-based AI technologies. GitLost exemplifies how the agent's context serves as both its operational basis and its vulnerability surface, similar to how SQL injections have historically impacted web security. To mitigate such vulnerabilities, Noma Labs recommends that organizations avoid treating user-controlled content as trusted input, restrict agent permissions, and sanitize inputs before processing. The responsible disclosure of this vulnerability to GitHub emphasizes the urgent need for enhanced security measures in AI systems.