Target Policy Optimization (arxiv.org)

A new approach in reinforcement learning, called Target Policy Optimization (TPO), has been introduced to enhance the policy update process in machine learning models. Traditionally, policy gradient methods simultaneously determine which model outputs to favor and how to adjust parameters, which can lead to inefficiencies due to overshooting or undershooting based on hyperparameter choices. TPO addresses this by distinctly separating these concerns; it constructs a target distribution based on the scored completions and fits the policy using cross-entropy. This leads to a more stable convergence as the loss gradient diminishes when the policy aligns with the target. The significance of TPO for the AI/ML community lies in its improved performance on tasks with sparse rewards, where it outperforms established methods like PG, PPO, GRPO, and DG while maintaining competitive results on simpler tasks. This method has been tested on various applications, including tabular bandits and transformer sequence tasks, offering promising results, particularly for large-scale language models. By providing a more refined way to optimize policies, TPO could lead to more efficient and effective reinforcement learning systems, thereby advancing the field's capabilities in complex decision-making scenarios.