Model hallucinations aren't random. They have geometric structure (arxiv.org)

Recent research has unveiled the Semantic Grounding Index (SGI), a novel metric that explores the geometric structure behind hallucinations in retrieval-augmented generation (RAG) systems. SGI quantifies the angular relationships between questions and generated responses in embedding space, revealing a phenomenon termed "semantic laziness." This means that when these systems generate hallucinated outputs, they tend to stay closer in angle to the original questions rather than drifting toward the retrieved context, emphasized by robust empirical results across various embedding models. This discovery holds significant implications for the AI/ML community, particularly for improving the reliability of RAG systems. The research indicates that SGI could serve as an efficient tool for identifying responses that require verification, especially in production deployments. Its calibration indicates that SGI scores can also offer probabilistic estimates of response accuracy, rather than merely ranking them. With the potential to enhance response validation mechanisms, SGI may play a crucial role in reducing misinformation generated by AI, particularly as it proves effective across varying contexts and question complexities.