Large Language Model Reasoning Failures (arxiv.org)

Recent research highlights significant reasoning failures in Large Language Models (LLMs), a critical issue despite their remarkable capabilities across various tasks. A comprehensive survey proposes a novel categorization framework that divides reasoning into two main types: embodied and non-embodied, with the latter further split into informal (intuitive) and formal (logical) reasoning. The survey also classifies reasoning failures into three categories: fundamental issues inherent to LLM architectures that impact a wide range of tasks, application-specific limitations affecting particular domains, and robustness challenges that lead to inconsistent performance with slight variations. This in-depth examination is significant for the AI/ML community as it sheds light on systemic weaknesses within LLMs and offers structured insights for future research. By defining and analyzing each type of reasoning failure and providing practical mitigation strategies, the authors aim to guide developers and researchers towards creating more robust and reliable reasoning capabilities in LLMs. Additionally, the release of a comprehensive GitHub repository containing related research serves as a valuable resource for those looking to explore this critical area further.