Large Language Models Struggle to Learn Long-Tail Knowledge (2023) (arxiv.org)

Recent research reveals that large language models (LLMs) face significant challenges in acquiring "long-tail" knowledge—information that is less frequently encountered on the web. The study highlights that a language model's ability to accurately respond to fact-based questions strongly correlates with the number of relevant documents encountered during its pre-training phase. By analyzing connection patterns in pre-training datasets, researchers found that while larger models demonstrate improved capabilities in learning this long-tail knowledge, substantial scaling is still necessary for them to perform competitively on less common questions. This finding is critical for the AI/ML community as it underscores the limitations of current LLMs in understanding infrequently represented knowledge. The research suggests that enhancing these models might require not only increasing their size but also implementing retrieval-augmented methods to alleviate their reliance on pre-training data. This approach could enable LLMs to better grasp rare pieces of information, ultimately improving their performance on a diverse array of question-answering tasks and enriching their overall utility in knowledge-intensive applications.