Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? (2024) (arxiv.org)

The paper introduces LOFT, a benchmark that tests long-context language models (LCLMs) on real-world tasks requiring access to corpora up to millions of tokens, to ask whether LCLMs can replace external tools like retrieval systems, retrieval-augmented generation (RAG), and even SQL-style database queries. By feeding whole corpora into a single model and evaluating in-context retrieval and reasoning, the authors show LCLMs—despite not being explicitly trained for these tasks—can surprisingly rival state-of-the-art retrieval and RAG pipelines on many tasks. Key strengths include simpler end-to-end workflows (fewer cascading errors), better user ergonomics (no special tooling), and the ability to apply global prompting strategies across an entire dataset. However, the study also surfaces clear limits: LCLMs struggle with compositional and multi-step reasoning tasks central to SQL-like queries, and their performance is highly sensitive to prompting choices. Practically, this suggests LCLMs could disrupt how we architect information systems—reducing reliance on index+retrieve pipelines—while raising trade-offs around compute, memory, and reliability for large-context inference. LOFT provides a rigorous testbed showing that as context windows grow, continued work on prompting methods, model architectures, and reasoning capabilities will be critical before LCLMs can fully subsume retrieval, RAG, and structured-query systems.