What stops a small language model from driving a database agent (arxiv.org)

Researchers have conducted an extensive study to evaluate the performance of small open-weight language models in driving a database agent. Over an 11-day period, they tested 39 models against an open-source SQL client, performing 8,199 operations and analyzing over 110,000 events. The findings revealed a significant limitation: 75.7% of agent-mode losses occurred during runs that invoked at least one tool, indicating that the models struggle with complex tasks involving external tools. The major failure type identified, dubbed “transport” failures, accounted for 36.2% of these losses, highlighting specific mechanical argument shapes that cause breakdowns. These results challenge the assumption that smaller models are fundamentally incapable of executing agentic database tasks. By pinpointing the causes of failures—ranging from server defects to the handling of argument shapes—the study emphasizes the importance of context and the tool interaction in model performance. Additionally, the researchers discovered a potential confound in local-model benchmarks related to computational resource limits, illustrating the need for a reevaluation of how such models are tested. The authors shared their dataset and tools, contributing valuable insights to enhance further research in the AI and ML communities.