Show HN: AgentLeak – open-source privacy testing for multi-agent LLM systems (github.com)

AgentLeak has been introduced as an open-source tool aimed at assessing privacy leakage in multi-agent large language model (LLM) systems. This benchmark serves as a practical resource for researchers and developers, providing a Python SDK and a comprehensive testing suite to identify sensitive data disclosures across various internal channels, including inter-agent messages, logs, and tool outputs. Published alongside an IEEE Access paper, AgentLeak evaluates 1,000 scenarios in domains such as healthcare and finance, revealing significant privacy vulnerabilities where internal channels exhibit leakage rates up to 96%, while external output audits miss nearly 46% of these violations. This development is crucial for the AI/ML community as it highlights the need for robust privacy measures when deploying multi-agent systems. The findings stress the alarming disparity in leakage rates between internal channels and final outputs, indicating that sensitive data can often be compromised even if the final results appear sanitized. Users can explore the tool’s capabilities through straightforward commands, allowing for tailored privacy assessments. By facilitating deeper insights into channel-specific vulnerabilities, AgentLeak underscores the importance of rigorous privacy testing in advancing safe LLM applications.