Leaderboard Hacking: Preference-Based Model Evals Vulnerable to Manipulation (arxiv.org)

Recent research highlights significant vulnerabilities in preference-based evaluation leaderboards for AI language models, which utilize pairwise comparisons to rank model performance. This study reveals that such rankings, commonly derived using the Bradley-Terry model, can be easily manipulated through strategies like strategic voting and prompting. These tactics enable unscrupulous developers to artificially inflate their models' standings by choosing biased prompts or casting irrelevant votes, potentially advancing a model's rank by as much as 45 places. This revelation is crucial for the AI/ML community, as it calls into question the integrity of widely-used model evaluation metrics. The research not only exposes the flaws in current ranking methodologies but also lays the groundwork for improving them. By understanding these manipulative tactics and their implications, the community can begin to develop measures that safeguard against such vulnerabilities, ensuring more reliable assessments of model performance moving forward. This scrutiny could lead to the adoption of stricter guidelines and methodologies, fostering greater trust in AI evaluations.