The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (arxiv.org)

Researchers have unveiled a notable limitation in auto-regressive large language models (LLMs), termed the "Reversal Curse." This phenomenon indicates that if a model is trained on phrases structured as "A is B," it fails to generalize the reverse structure "B is A." For instance, a model trained on the statement "Valentina Tereshkova was the first woman to travel to space" cannot successfully answer, "Who was the first woman to travel to space?" This finding reveals a critical gap in how these models understand and relay information, undermining their presumed capability to apply learned patterns. Significantly, the Reversal Curse persists across various model architectures and sizes, including GPT-3, Llama-1, and ChatGPT, irrespective of data augmentation efforts. In practical terms, this limitation impacts the accuracy of the models in real-world applications, where users may expect robust knowledge retrieval. For example, while GPT-4 performs accurately in 79% of straightforward questions about celebrity relationships, it only achieves 33% accuracy on reversed inquiries. This research sheds light on the complexities of language understanding in LLMs and highlights the need for improved strategies in model training to ensure comprehensive comprehension and retrieval capabilities.