Agent Learning via Early Experience (arxiv.org)

Researchers propose "early experience," a middle-ground training paradigm for language agents that uses the agent’s own interaction data—specifically future states reached by the agent—as supervision even when scalar rewards are unavailable. They study two concrete strategies: implicit world modeling, where collected states help ground the policy in environment dynamics (a form of learned transition or predictive model), and self-reflection, where agents learn from their own suboptimal actions to improve chain-of-thought reasoning and decision-making. This sidesteps limits of supervised fine-tuning on narrow expert demonstrations and the inefficiency of long-horizon reinforcement learning rollouts in environments like websites or multi-turn tool use. Across eight diverse environments and multiple model families, early experience consistently boosted task effectiveness and out-of-distribution generalization, and in reward-bearing environments it produced promising foundations for later RL fine-tuning. The work is significant because it offers a scalable, architecture-agnostic way to increase environment diversity exposure, improve sample efficiency, and bridge imitation learning and fully experience-driven agents—making it easier to train robust, adaptable language agents in real-world settings where rewards are sparse, delayed, or hard to verify.