Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation (arxiv.org)

Researchers have unveiled LivingArena, a novel evaluation framework designed to assess large language models (LLMs) by exploring their knowledge boundaries. Traditional benchmarking methods face challenges with contamination and saturation, which hinder the ability to differentiate between top-performing models or understand specific weaknesses. LivingArena introduces a dynamic peer-probing approach where LLMs alternate in posing questions aimed at identifying what their counterparts cannot answer. By leveraging this competitive setting, the framework not only rewards effective questioning but also employs a panel of strong models to ensure the validity of the questions posed. This development is significant for the AI/ML community as it presents a scalable, low-cost method for continuous model evaluation, moving beyond static knowledge recall. The evaluation results created an Elo leaderboard for ten frontier LLMs, revealing insights into how models can strategically exploit each other’s weaknesses. The findings also show that peer probing correlates weakly with human preferences, hinting at the potential need for more nuanced evaluation criteria. Overall, LivingArena represents a step forward in understanding LLM capabilities, offering a fresh perspective on model assessment and fostering a deeper comprehension of cognitive dimensions in AI systems.