Discretizing Reward Models (arxiv.org)

Researchers have unveiled significant insights into the limitations of reward models in reinforcement learning, revealing that many popular models are oversensitive in assigning continuous scores to responses, leading to inconsistencies and ineffective policies. This study highlights the need for better evaluation criteria for reward models, proposing "discriminative ability" and "specificity" as alternatives to traditional notions of accuracy. To address these challenges, the researchers introduce a training-free algorithm leveraging Monte Carlo dropout to produce discrete reward clusters, which effectively mitigates oversensitivity while maintaining a strong capacity for discrimination. Their empirical findings demonstrate that discretizing rewards can reduce instances of reward hacking and enhance the performance of reinforcement learning policies, offering a promising direction for future applications in AI/ML that demand more reliable and robust evaluation mechanisms.