Public LLM benchmarks are mostly garbage (grandpacad.com)

A recent evaluation of several leading 3D code generation models revealed that public benchmarks are often misleading, particularly in how they assess real-world performance. Testing models such as Opus 4.7, Gemini 3.1, GPT 5.5, and Kimi K2.6 against real user prompts demonstrated that Opus 4.7 significantly outperformed its competitors in terms of speed, cost, and effectiveness, scoring a weighted rating of 0.587 and completing tasks in an average of just 32 seconds at a cost of $0.10 per generation. In contrast, GPT 5.5 and Gemini were slower and more expensive while not providing a substantial advantage in accuracy. This finding is crucial for the AI/ML community as it underscores the importance of contextually relevant benchmarks over generic ones that may favor models based on subjective metrics like render quality. The study highlights the jagged nature of frontier model performance, where different tasks reveal varying capabilities, and emphasizes the necessity of tailored evaluations. The message is clear: real-world testing with specific use cases is essential to determine a model's true usability, especially in specialized domains like 3D modeling. As a result, many users might need to reconsider reliance on public benchmarks and adopt more rigorous testing methodologies tailored to their specific applications.