64 Tiny Benchmarks for Jev (www.ramonov.com)

A recent exploratory analysis on Jev, a new AI model positioned between traditional large language models (LLMs) and specialized classifiers, has uncovered intriguing performance patterns through 64 distinct benchmarks. The analyst conducted tests by repeatedly submitting questions—both true/false and multiple-choice—50 times each, focusing on the model's output distributions instead of single responses. The study revealed inconsistencies, such as contradictory answers regarding the count of the letter 'r' in "strawberry" and notable variances based on character representation in questions. Such findings indicate that while Jev can maintain high accuracy, there are blind spots where it struggles with logic and consistency. These insights are significant for the AI/ML community as they highlight the potential and limitations of Jev in classification tasks. Its unique ability to provide probability distributions over options pushes the boundaries of traditional AI response models, making it suitable for various applications yet susceptible to peculiarities in input. The analyst’s exploration serves as a reminder of the importance of rigorous evaluation to understand how such systems respond to nuanced queries, paving the way for further advancements and refinements in AI design.