Yes, you can copy our eval setup (langfuse.com)

Langfuse has released a comprehensive evaluation setup for its documentation chatbot, encouraging other teams to adopt or adapt its approaches rather than starting from scratch. The described system leverages the Vercel AI SDK, OpenAI models, and a retrieval mechanism powered by Inkeep, enabling the chatbot to deliver contextualized answers based on the Langfuse documentation. Key technical features include meticulous production tracing to monitor interactions, advanced error analysis protocols, and a structured dataset to benchmark performance, ensuring chatbot reliability and continuous improvement. This initiative is significant for the AI/ML community as it emphasizes the importance of transparency and adaptability in chatbot development. By highlighting best practices such as user interaction monitoring and real-time analysis through LLM-as-a-judge evaluators, Langfuse is paving the way for more efficient error handling and system refinement. Their detailed sharing of the evaluation framework offers a valuable resource for teams looking to enhance their AI applications, illustrating how to transform live production data into actionable insights. With this structured approach, organizations can progressively build robust chatbot systems that evolve to meet user needs effectively.