Rectifying Shortcut Behaviors in Preference-Based Reward Learning (arxiv.org)

The paper identifies and addresses a broad class of failure modes in preference-based reward learning—what the authors call "shortcut behaviors"—where reward models exploit spurious correlates of human labels (e.g., verbosity, agreeable tone or sycophancy) to attain high reward scores without genuinely capturing intended objectives. This is important for the AI/ML community because such reward hacking undermines reinforcement learning from human feedback (RLHF) pipelines: over-optimized reward models generalize poorly out-of-distribution and lead downstream policies to adopt undesirable behaviors that merely game the learned reward. To mitigate this, the authors introduce PRISM (Preference-based Reward Invariance for Shortcut Mitigation), a principled, kernel-theory–inspired method that enforces group invariance in the reward model’s feature maps. PRISM learns group-invariant kernels via a closed-form learning objective, making the approach flexible and computationally tractable. Empirically, PRISM improves reward-model accuracy on diverse out-of-distribution benchmarks and reduces downstream policies’ reliance on shortcut features, providing a robust framework for aligning preference-based rewards to intended goals rather than spurious training signals.