Show HN: verbatimeter - check how grounded your LLM / RAG agent is in real-time (pypi.org)

A new tool called "verbatimeter" has been introduced, enabling developers to deterministically assess how grounded AI-generated text is in its source material. This tool highlights verbatim and paraphrased text, counts differing tokens, and can gate pipelines based on the accuracy of quotations without relying on a judge model or external networks. The integration is designed to fit seamlessly into existing retrieval-augmented generation (RAG) agents, allowing users to check how much of the generated content directly correlates to the source text. The significance of verbatimeter lies in its potential to enhance transparency and reliability in AI outputs, particularly in applications where accuracy is critical, such as journalism or academic research. By using a decorator or command-line interface, developers can verify the groundedness of their models in real-time, ensuring that generated responses conform closely to the sourced information. This is crucial for identifying hallucinations or fabricated content, making verbatimeter a valuable resource in combating misinformation and enhancing the trustworthiness of AI-generated text across multiple languages.