The LLM Looked Smart. The Metrics Disagreed (tiago.rio.br)

In a recent case highlighted from 2025, a bank faced significant backlash after expanding access to free digital accounts for customers without credit. Despite early optimism from using a large language model (LLM) to classify customer complaints, the results revealed a stark discrepancy between perceived effectiveness and actual performance, with a recall rate as low as 42%. This situation underscored the limitations of relying solely on LLMs and prompted a reevaluation of the approach to customer feedback. To address the issue, the team initially relied on prompt engineering but soon realized that a more robust classification model was needed. After manually labeling thousands of reviews, they developed a more accurate classifier using XGBoost with BERT text embeddings, which improved precision to 81% and recall to 65%. However, it wasn't until they fine-tuned an OpenAI GPT model specifically for the classification task that they achieved an impressive recall of 86% and precision of 91%. This experience illustrates the importance of rigorous evaluation and the value of fine-tuning models to achieve reliable results in real-world applications, proving that while LLMs can assist, they may not always provide the complete solution.