ToTra – open-source LLM gateway with GDPR/EU AI Act compliance (github.com)

ToTra, an innovative open-source AI gateway and governance platform, has been launched to enhance compliance and management for large language model (LLM) usage. Written in Go, it provides seamless integration with existing apps by requiring just a one-line code change to direct requests through ToTra instead of directly to providers like OpenAI or Anthropic. Key features include per-user and per-team budget caps, real-time personal identifiable information (PII) scanning across 18 language groups, comprehensive cost tracking, and built-in compliance with GDPR and the EU AI Act, all while maintaining under 2 ms latency at the 95th percentile. This release is significant for the AI/ML community as it addresses pressing concerns regarding data privacy, compliance, and financial oversight in the growing landscape of LLM applications. The gateway's ability to block sensitive data before it reaches external networks reinforces data security, while its robust auditing and reporting capabilities help organizations adhere to stringent regulatory standards. Furthermore, its self-hosted nature ensures that user data remains under the organization's control, thereby facilitating a safer and more regulated AI deployment environment.