Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory (arxiv.org)

A new approach to enhancing long-horizon tasks in large language models (LLMs) has been introduced through a mechanism called Memex, which utilizes indexed experience memory. Traditional methods of managing the context in LLMs often lead to information loss due to truncation or summarization, making it challenging to leverage past interactions effectively. Memex addresses this by maintaining a compact but structured working context combined with an external database that retains full-fidelity interactions. By allowing the agent to dereference these indices as needed, Memex enables a flexible and efficient method of recalling past evidence, significantly improving task performance while minimizing context size. The significance of Memex lies in its reinforcement learning framework, MemexRL, which optimizes the processes of writing, summarizing, archiving, indexing, and retrieving information within a fixed context budget. This not only refines how LLM agents manage long-term memory but also ensures that decision quality remains high even as complexity grows. The preliminary empirical results highlight that agents trained with MemexRL perform better on challenging long-horizon tasks, paving the way for more effective applications of LLMs in scenarios demanding extensive contextual knowledge and reasoning.