The AI Disagreement Index: 8 models agreed on the "best tool" 0 of 16 times (data.deepsynthesis.org)

The AI Disagreement Index has been launched, showcasing the divergence in recommendations among multiple AI engines when queried about the "best tool" for specific categories. Unlike traditional rankings that offer a generic "share of voice," this new index focuses on cross-engine disagreement, revealing how various models prioritize different tools. Notably, across its analysis, the index found that while 8 models consistently identified the "best tool," they did not agree on this choice even once across 16 categories, highlighting the varying perspectives each AI system brings to the table. This index is significant for the AI/ML community as it shifts the focus from mere brand rankings to nuanced insights into model behavior and decision-making. It employs rigorous methodologies, including Fleiss' kappa for consensus measurement, and all data is openly published in a reproducible format. Monthly updates allow for tracking of changes in AI recommendations over time, promoting transparency and encouraging a deeper understanding of how these models evolve. By revealing the complexities of AI recommendations, the Disagreement Index serves as a critical tool for organizations looking to navigate a crowded marketplace while fostering accountability in AI-driven insights.