How Confident Are AI Classifiers About Their Own Confidence? (gmcirco.github.io)

Recent explorations into AI classifiers have raised important questions about how accurately these models can assess their own confidence in classifications. The piece discusses the challenges of deriving reliable confidence probabilities from Large Language Models (LLMs) compared to traditional machine learning techniques, emphasizing that they lack a straightforward method for obtaining confidence scores. Instead, practitioners typically either prompt the LLMs for their confidence or analyze token-level probabilities directly. The author tested an LLM-based extraction pipeline on medical narratives from the National Electronic Injury Surveillance System (NEISS), achieving a classification accuracy of 86%, but noted potential calibration issues in confidence scores. The implications for the AI/ML community are significant. The findings underscore the need for improved calibration methods to ensure that AI-generated confidence scores align more closely with actual classification accuracy. Instead of solely relying on the inherent probabilities provided by LLMs, the piece suggests a "top versus all" calibration approach using isotonic regression to adjust these scores more effectively. This refinement could enhance the reliability of AI systems in critical applications like healthcare, where understanding the certainty of classifications is paramount for decision-making.