I hope to help you evaluate your GenAI App (github.com)

Evalyn has introduced a local-first framework designed to simplify the evaluation and calibration of Generative AI (GenAI) applications, making it accessible for both developers and non-technical users. This framework allows all data to remain on the user’s machine without cloud dependencies, utilizing SQLite for storage. Key features include auto-captured LLM calls, a toolkit of over 50 metrics for quality assessment, and automated calibration processes that align LLM judges with human feedback through prompt optimization techniques. Users can initiate a comprehensive evaluation pipeline with a single command, or choose a step-by-step approach for granular control. The significance of Evalyn lies in its democratization of GenAI app evaluation, encouraging continuous improvement by helping users gauge app performance against real-world usage. The framework's emphasis on local processing and a variety of readily available evaluation metrics allows teams to iteratively assess and enhance their models, making it particularly valuable in a landscape where rapid iteration and user feedback are crucial. This initiative not only streamlines the assessment process but also provides practical tools to refine LLM behaviors based on user interaction, marking a notable advancement in the AI/ML community's ongoing quest for effective and user-friendly evaluation methodologies.