Show HN: NullPrime – do AI models know when they can't know? (github.com)

A recent project named NullPrime has unveiled significant insights into the limitations of AI models, specifically in their ability to recognize when they lack knowledge about certain tasks. The research introduced a unique dataset of integers, labeled as prime or composite, where the normal indicators of primality are intentionally obscured. Notably, this study found that most AI models, including popular large language models (LLMs), fail to indicate uncertainty correctly, often expressing confidence in incorrect answers instead of admitting they cannot determine the primality of a number. This work is crucial for the AI/ML community as it serves as a controlled benchmark for evaluating uncertainty in predictive models. By stripping away learnable features related to primality, NullPrime isolates the inability of these models to gauge their own knowledge limits. The findings highlight the importance of developing robust frameworks for uncertainty estimation, as many AI systems demonstrated a propensity to make incorrect assertions with high confidence. The detailed results reveal that while some models can achieve high accuracy with straightforward questions, they struggle significantly with complex or nuanced queries, emphasizing the need for improved AI transparency and reliability in decision-making processes.