Forget about credit assignment in long-horizon agentic post-training (qiyaoliang.com)

A new analysis in the AI/ML field highlights the advantages of Evolution Strategies (ES) as a robust alternative to Policy Gradient (PG) methods for long-horizon agentic post-training tasks. Unlike PG, which struggles with "credit assignment" issues—where every action's impact muddles learning over lengthy sequences—ES shows promise by bypassing this complexity. Research indicates that ES efficiently fine-tunes billion-parameter models without suffering from the traditionally associated "curse of dimensionality." Preliminary results demonstrate that finetuning with ES on the Qwen3.5-4B model outperforms the PG technique on challenging real-world benchmarks. The significance of this finding lies in the potential for simpler and more effective training methodologies in reinforcement learning, particularly for long-agent tasks where past methods faltered due to their complexity. As ES proves to be embarrassingly parallelizable and requires no intricate engineering around credit assignment, it paves the way for revisiting zeroth-order optimization methods, suggesting a paradigm shift in training strategies that could enhance sample efficiency in AI development moving forward. This shift underscores the changing dynamics in AI research, potentially altering how large models are trained and optimized in future applications.