Contrakit: Predicting Model Hallucination Before Training (github.com)

Contrakit recently announced a novel approach to predict model hallucination before training, shedding light on a persistent challenge within AI and machine learning. Hallucination, where models generate confident but inaccurate outputs, primarily arises from structural and architectural issues in neural networks. The research identifies two key sources of hallucination: structural incompatibilities in task definitions and the limitations imposed by the softmax architecture, which forces models to produce definitive predictions even when context is unclear. This insight is critical as it suggests that merely increasing training data won't resolve these underlying issues. The findings reveal that the complexity of tasks can be quantified using a metric called K, measuring intrinsic contradictions in the task. When K is greater than zero, some error becomes mathematically unavoidable. Additionally, standard softmax architectures struggle with expressing uncertainty, as they offer no mechanism for models to indicate when they lack sufficient information. The research emphasizes the importance of implementing architectural adaptations that allow for context-aware abstention in order to effectively mitigate hallucination, representing a significant shift in how the AI/ML community approaches model training and output reliability.