Artificial Analysis Decision Index v1.0 (artificialanalysis.ai)

The newly launched Artificial Analysis Decision Index (AADI) v1.0 introduces a comprehensive benchmarking tool designed for evaluating decision models, a novel category of AI that specializes in providing structured answers with associated probabilities through a single call. This index incorporates 500 tailored questions from existing knowledge and hallucination benchmarks, alongside other evaluations focused on classification, tool calling, and reasoning, offering insights into model accuracy, speed, and cost. With 25% weight assigned to various performance metrics, the index serves as a crucial resource for developers and companies seeking to navigate the rapidly evolving landscape of decision-making AI models. This benchmark is significant for the AI/ML community as it highlights the emerging capabilities of decision models, which promise lower costs and faster responses compared to traditional language models. Notable contenders include Jev 1.13, which leads in reasoning but has room for improvement in knowledge representation. Costs for decision models range widely, from $17 to $199 per million tasks, while calibration issues persist, particularly with knowledge and reasoning tasks. With decision models showing competitive performance in structured tasks yet trailing language models in knowledge-based accuracy, the AADI v1.0 offers a clear roadmap for advancements in AI decision-making and its practical applications across industries.