Potential-Based Reward Shaping in Reinforcement Learning (medium.com)

Recent advancements in reinforcement learning (RL) have introduced potential-based reward shaping, a method that accelerates the training of agents by offering additional guidance without altering their optimal strategies. Traditionally, agents like a soccer-playing robot only receive a reward at the end of a task—scoring a goal—making it difficult for them to learn effectively. By employing a potential function, reward shaping offers incremental rewards based on the agent's progress toward its goal, allowing for more frequent and informative feedback. In mathematical terms, the shaped reward is defined as R′(s, a, s′) = R(s, a) + γ·Ψ(s′) − Ψ(s), where Ψ(s) assesses the desirability of a state. This method ensures that while agents gain essential motivation to transition toward better states, it cleverly cancels out over time, preserving the original learning dynamics. An illustrative example includes a potential function reflecting the distance to a soccer goal, rewarding the robot for moving the ball closer. This innovative approach not only enhances computational efficiency but also provides agents with a smarter learning starting point, maintaining convergence to optimal policies without compromising performance.