What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models (arxiv.org)

A recent study examined the epistemic diversity of large language models (LLMs), which refers to the range of real-world knowledge reflected in their outputs. Despite the prevailing assumption that these models lack diversity, this research provides the first comprehensive assessment across time and cultural contexts. Testing 27 LLMs on 155 topics spanning 12 countries generated over 1.7 million responses, revealing that epistemic diversity has significantly improved in the past three years. However, the study also found that all LLMs exhibited less diversity compared to a search engine baseline, indicating that while there is progress, it is insufficient and inconsistent. The findings highlight critical implications for the AI/ML community. Larger models were found to be counterintuitively less diverse than smaller ones, and their knowledge predominantly reflects English-language contexts, sidelining country-specific insights. This uneven distribution of epistemic diversity could challenge the robustness of LLMs as knowledge sources, potentially leading to a homogenization of information. Researchers argue for enhanced diversity measurement techniques and methodologies to ensure that AI-generated content reflects a broad spectrum of global knowledge.