Show HN: I built a deterministic check for fabricated quotes in LLM output (github.com)

A new tool called "verbatimeter" has been introduced to address the issue of verifying the groundedness of responses generated by large language models (LLMs). This tool offers deterministic checks for fabricated quotes in LLM outputs by measuring both verbatim reuse and paraphrasing. Users can easily integrate it into retrieval-augmented generation (RAG) systems using a simple decorator or command-line interface (CLI). Noteworthy features include the ability to distinguish between verbatim passages and the model's own wording, providing clear visual feedback in real time. For instance, it can highlight matched text in green while marking model-generated wording in red, allowing for precise verification of quoted material. The significance of verbatimeter lies in its potential to enhance the reliability of LLM outputs, particularly in critical applications where citation accuracy is essential. By accurately quantifying the extent of text sourced from original documents and identifying fabrications, this tool serves as a valuable resource for researchers and practitioners in the AI/ML community. It supports multilingual text and operates offline with minimal dependencies, making it an accessible option for a diverse range of users. The deterministic nature of the checks ensures consistent results, which can help prevent the propagation of misinformation in AI-generated content.