Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting (arxiv.org)

Researchers introduced ECHO (Experience Consolidation via Hindsight Optimization), a prompting framework that makes language-model (LM) agents far more sample-efficient by transforming failed interactions into synthetic successes. Drawing inspiration from hindsight experience replay in reinforcement learning, ECHO uses the LM itself to infer alternative goals achieved within an episode and then generates optimized counterfactual trajectories for those goals. The method has two parts: a hindsight rule that identifies relevant subgoals and rewrites trajectories as if those subgoals were intentionally pursued, and an update rule that stores compressed trajectory representations in memory for efficient retrieval and reuse. ECHO is significant because it leverages LMs’ generative and reasoning abilities to create full counterfactual episodes, reducing the need for costly real-world interactions or environment resets. In evaluations on stateful XMiniGrid (text-based navigation/planning) and PeopleJoinQA (collaborative information gathering), ECHO outperformed vanilla LM agents by up to 80% and beat advanced architectures like Reflexion and AWM in XMiniGrid, showing faster adaptation to novel environments. Technically, the approach demonstrates that hindsight trajectory rewriting plus compressed episodic memory can convert failures into valuable training data, improving online learning efficiency and suggesting a practical route to safer, cheaper deployment of LM agents in human-in-the-loop and physical settings.