Show HN: UQLM – Closed-book hallucination detection with UQ (github.com)

The recently introduced UQLM library is a powerful Python toolkit designed for detecting hallucinations in outputs from Large Language Models (LLMs) using advanced uncertainty quantification techniques. This new version, which can be easily installed via PyPI, provides a range of response-level scorers that quantify the uncertainty of LLM outputs, yielding confidence scores from 0 to 1. The library features several types of scorers, including black-box, white-box, and LLM-as-a-Judge scorers, catering to various operational needs related to cost and latency. Notably, UQLM can be utilized universally with any LLM, making it a highly adaptable tool for researchers and developers. The significance of UQLM lies in its ability to systematically assess and mitigate errors associated with LLM outputs, thereby enhancing the reliability of AI-generated content. The toolkit’s approach not only includes multi-response generation but also allows for fine-grained evaluation of individual claims within long-form text, ultimately fostering more accurate and trustworthy AI applications. With detailed documentation and example notebooks for various quantification techniques, UQLM empowers the AI/ML community to improve the robustness of LLMs while addressing the critical challenge of hallucination detection in AI communications.