Why AI Benchmarks Are Total BS (www.pcmag.com)

Recent scrutiny of AI benchmarks reveals significant flaws in their reliability and relevance, particularly as companies like OpenAI and Anthropic promote their models' scores. Benchmarks such as Terminal-Bench and GeneBench, often cited as performance indicators for models like GPT-5.6 and Opus 5, may not accurately reflect their real-world utility. The evaluation of these models can fluctuate dramatically from one version of a benchmark to the next, creating confusion around their true capabilities. For instance, GPT-5.6 initially outperformed Fable 5 before later versions of Terminal-Bench showed the opposite trend. This inconsistency suggests that the benchmarks may not be effective indicators of a model’s performance in practical applications. The inherent conflict of interest arising from AI companies creating and administering their benchmarks amplifies concerns about transparency and trust. As AI models are versatile tools, measuring their effectiveness using specific benchmarks often fails to capture their overall intelligence and adaptability. This situation not only complicates the decision-making process for developers and users but also highlights the need for more standardized, independent evaluation metrics within the AI/ML community. As the industry develops, understanding these benchmarks' limitations will be crucial for evaluating AI models authentically.