When LLMs Fail at Classification (jeffyclassify.com)

A new open-source text classifier named Jeffy has emerged, demonstrating superior performance in multi-class classification tasks compared to various language models (LLMs) and decision models. By utilizing BGE-large embeddings and logistic regression, Jeffy was benchmarked against 15 different models, including local LLMs ranging from 0.5B to 32B parameters and other classifiers like SetFit and FastText. The results reveal that while initial tests with two to four classes show competitive performance across models, Jeffy's accuracy significantly outshines the rest as the number of classes increases, particularly when classifying up to 77 categories. This breakthrough highlights the limitations of LLMs and decision models in handling complex, high-dimensional classification tasks. As the number of classes grows, LLMs struggle with overlapping semantic meanings, leading to confusion and poor performance. In contrast, Jeffy effectively navigates these challenges by leveraging its training on labeled data to understand the boundaries between classes, achieving lower latency and greater accuracy without relying on external APIs or significant hardware. With a compact model size of under 5MB, Jeffy offers an efficient and private solution for developers and researchers needing a reliable classifier.