Where Jev worked for us, and where it didn't (gitloom.ai)

TypeSafe recently unveiled Jev, a new AI model designed to improve decision-making processes by providing fixed answer types (yes/no or predefined options) along with a confidence score. This model was integrated into GitLoom, a tool that summarizes GitHub team's activities and makes hundreds of small decisions daily about pull requests. Jev's main advantage lies in its ability to convey uncertainty through its confidence number, allowing GitLoom to act only on high-confidence responses, thus maintaining operational efficiency. The model has shown impressive performance, achieving a 98% accuracy rate on categorized cases, which is only slightly below the traditional method it replaced. However, Jev's implementation revealed limitations, particularly in ranking relevance. While it effectively identifies whether a pull request relates to a query, it does not distinguish the importance of multiple relevant results. As a result, the model could assign similar high confidence scores to pull requests of varying significance, necessitating further experimentation to refine this aspect. For GitLoom users, the integration of Jev has transformed output by providing more insightful summaries and enabling flexible topic queries, enhancing the tool's overall functionality. Users can experience GitLoom with a free 10-day trial, simplifying setup and delivering immediate insights into their repositories.