Don't ask an LLM for a confidence score (justinflick.com)

Recent discussions within the AI community have highlighted a crucial flaw in the use of confidence scores generated by large language models (LLMs). An expert argues that these scores, often formatted as a continuous scale from 0 to 100, lack scientific validity and can create a false sense of trustworthiness. The critique emphasizes that LLMs are not reliably equipped to assess their own correctness, pointing out that existing research shows models can recognize some aspects of their outputs but do so in a context-dependent and unreliable manner. Consequently, prompting an LLM to provide a confidence score may do more harm than good, as it conflates ambiguous metrics of correctness, coherence, and response intent into a single number. The implications for the AI/ML community are significant. Without a robust methodology for quantifying confidence, the default reliance on a continuous score may mislead users about the reliability of model outputs. The piece advocates for categorizing confidence metrics rather than relying on continuous scales, as the latter has been shown to cluster responses around arbitrary anchors rather than reflect true probabilistic understanding. Ultimately, researchers and developers are encouraged to pursue more nuanced approaches, incorporating calibration techniques and human-defined heuristics rather than accepting self-reported confidence as a credible measure of performance.