Using LLM for confidence scoring is a bad idea (rumple.github.io)

A recent blog post argues that using large language models (LLMs) to generate confidence scores for predictions is fundamentally flawed. While LLMs can output confidence levels, the author stresses that these scores lack reliability because they are not based on concrete information or prior data about actual outcomes. Unlike a weatherman who can derive confidence from historical weather patterns and current conditions, LLMs generate scores purely based on the probabilistic continuation of text, leading to arbitrary and inconsistent outputs. For instance, the same question can yield different confidence scores upon rephrasing, undermining the consistency that is essential for trustworthy predictions. This critique highlights a significant concern for the AI/ML community as reliance on LLMs for confidence scoring can lead to overconfidence in automated systems, particularly in critical applications like customer support. Instead, the author advocates for a systematic approach to measuring confidence based on concrete, observed data. By creating a rule-based system that utilizes information about previous interactions and outcomes, developers can generate reliable confidence scores that evolve with increasing data, ensuring a more accurate reflection of prediction reliability and improving decision-making in AI applications.