Wandr Benchmark: Evaluating Research Agents That Must Search Wide and Deep (research.perplexity.ai)

Today, a benchmark named WANDR (Wide ANd Deep Research) has been released, aimed at evaluating AI research agents on complex data-collection tasks within the realm of knowledge work. With 500 realistic tasks such as competitive mapping and market analysis, WANDR complements the existing DRACO benchmark that focuses on generating comprehensive long-form reports. This new evaluation method emphasizes the dual requirements of wide (discovering numerous qualifying entities) and deep (investigating each entity thoroughly) research, highlighting the need for agents to not only find information but also substantiate it with aligned evidence. The significance of WANDR for the AI/ML community lies in its potential to bridge the gap between theoretical capabilities and real-world performance in research-intensive tasks. Early evaluations show that existing systems are struggling to achieve high precision and recall, with even the best performing system, Perplexity, only reaching a soft F1 score of 0.363 and a hard F1 score of 0.133. WANDR’s flexible structure, which allows for various configurations of data hierarchies, aims to stimulate advancements in how AI can effectively perform wide-and-deep research tasks. By providing a pathway to identify strengths and weaknesses in current systems, WANDR sets the stage for future improvements and innovations in AI research technology.