Show HN: Auditi – open-source LLM tracing and evaluation platform (github.com)

Auditi has launched an open-source platform designed for the evaluation and observability of AI agents and large language model (LLM) applications. This comprehensive tool offers features such as automatic trace capture, LLM-as-a-judge evaluations, and human annotation workflows that streamline the process of monitoring and improving AI systems. Auditi’s capabilities include detailed analytics and custom data management, allowing users to create reusable datasets for fine-tuning, cost tracking based on provider pricing, and failure mode analysis to generate actionable insights. The significance of Auditi for the AI/ML community lies in its potential to enhance agent performance assessment and integration flexibility. With support for major LLM providers like OpenAI, Anthropic, and Google Gemini, the platform promotes seamless integration through minimal code changes. Additionally, it accommodates complex workflows with features like async support and custom evaluators. As an open-source solution, Auditi fosters collaboration and innovation within the community, providing a valuable resource for developers aiming to build improved AI systems efficiently.