Comparing Jev, Gemini, and OpenAI decision models (blog.slopcop.com)

In a recent evaluation of decision models for AI agents, a bakeoff was conducted comparing Jev, Gemini Decisions, and GLiDE from Fastino Labs. Mjolnir, the agent platform, sought to enhance relevance detection for GitHub events, crucial for keeping teams informed about related developments. While Jev was the initial choice, the team ultimately selected GLiDE for its outstanding performance in handling relevant data while being one of the most cost-effective options. GLiDE’s open-ish weights model offers a flexible commercial use policy, although it was noted to be slower than Gemini and OpenAI’s offerings. The evaluation underscored significant lessons for the AI/ML community, particularly around the importance of independent validation of models rather than relying solely on incumbent verdicts. By providing judges with comprehensive context and avoiding biased sampling methods, the study demonstrated how to better assess model performance. The findings suggest that while Jev remains a viable option for those with limited resources, GLiDE provides a superior alternative for teams looking to maximize relevance detection in their projects, emphasizing the need for continuous improvement and informed decision-making in model selection.