1 in 4 3rd Party AI Endpoints Underperform Their Advertised Models (hermes-ai.net)

Artificial Analysis has released version 1.1 of its Endpoint Accuracy Index, revealing that nearly 25% of serverless AI endpoints do not perform to the standards of their advertised models. The index evaluates more than 100 endpoints against reference implementations using official weights and measures accuracy across three key capabilities: tool and function calling, reasoning, and long-context recall. The update narrows the confidence interval for accuracy scores, reducing uncertainty in comparisons between endpoints. With this release, the index now includes six open-weight models, including versions of GLM and DeepSeek, providing deeper insights into performance variance across providers. The findings underscore a significant challenge in the AI/ML community: differences in endpoint performance can stem from factors such as output limitations and reasoning capacities rather than just model specifications. For instance, some endpoints suffer from truncated outputs, dropping their accuracy on reasoning tasks considerably. This highlights the importance of rigorous evaluation metrics for companies selecting AI models, emphasizing the necessity of considering not only price and speed but also precise accuracy scores. Ultimately, the Endpoint Accuracy Index serves as a critical tool for guiding developers in their choices, ensuring they are aware of the potential pitfalls in using server-hosted AI models.