Our AI Agent Executed the Same Successful SQL Query 254 Times (armincat.com)

Funnel has conducted a detailed evaluation of their internal data agent platform, Reef, utilizing different language models, specifically GLM-5.2 and Claude Opus 4.8. The evaluation showed that while GLM-5.2 was quicker in individual model calls, averaging 7.8 seconds versus Opus's 18.1 seconds, it was less efficient in overall task execution, leading to a 14% longer completion time due to making 66% more tool calls. This discrepancy highlighted the importance of understanding not just the speed of model responses but also the effectiveness of their actions, as GLM executed the same SQL query multiple times without retrieving new data. The findings are significant for the AI/ML community, emphasizing that model performance cannot be isolated from the framework surrounding it. Issues such as unclear tool contracts, repetitive calls, and ambiguous instructions can lead to inefficiency, underscoring the need for explicit guidelines and constraints when integrating open-weight models into agentic systems. Funnel suggests several strategies for better model performance, including making tool contracts clear, governing agent behavior to prevent redundancy, and comprehensively benchmarking system performance beyond just raw model speed. These insights highlight the complexities of using AI in practical applications and the need for robust infrastructure to maximize the potential of advanced language models.