LLM gave you an answer. Should your application trust it? (github.com)

The newly announced BOOTH is a lightweight checkpoint layer designed to enhance the reliability of applications using large language models (LLMs). It bridges the gap between LLM calls and application outputs by providing structured, defensible decisions on whether the model's answers can be trusted. Rather than simply accepting or rejecting the LLM’s outputs, BOOTH evaluates the validity of responses based on evidence and predefined checks, ensuring that app developers can avoid delivering confidently incorrect information to users. This tool is significant for the AI/ML community as it addresses a common challenge: the discrepancy between LLM confidence and the correctness of the information provided. BOOTH incorporates features such as ambiguity detection, confidence checking, and a clear mechanism for evidence validation, all while maintaining a provider-agnostic approach. Developers can install it easily via pip and integrate it into their existing workflows, allowing for more robust applications that prioritize accuracy over mere fluency in LLM-generated text. The emphasis on structured outputs, retry logic for reconsideration, and detailed status reporting highlights BOOTH's potential to significantly improve the trustworthiness of AI-driven applications.