New Data Science LLM Benchmark (proud-botany-7dd.notion.site)

A new benchmark comparing large language models (LLMs) on practical data science tasks highlights the superiority of domain-specialized agents over general-purpose models. The study tested five LLMs—DGi, Replit Chat, OpenAI’s o3, Claude, and a baseline model v0—across four data-related tasks including CSV repair, deduplication, session analytics, and live endpoint metric queries. DGi emerged as the clear winner with a 33% higher overall score than the best generic model, thanks notably to its ability to ask clarifying questions and handle ambiguous schemas, a crucial advantage when working with poorly defined data structures. Technically, the evaluation used small synthetic datasets and a one-shot prompting methodology with no retries. Scoring combined row-level data overlap with analytic accuracy, demonstrating that models like OpenAI’s and Claude struggled especially on endpoint queries, failing to produce usable results without explicit guidance. DGi’s interactive clarification step enabled near-perfect endpoint predictions, affirming that resilience to vague prompts and schema inference differentiates specialized data agents from generic LLMs that plateau under such conditions. The benchmark also underscores current limitations, such as reliance on synthetic data and absence of chain-of-thought prompting, pointing toward future tests on more complex, noisy datasets and real-time performance metrics. This benchmark signals a growing need for AI models tailored to the nuances of data engineering workflows, where ambiguity and implicit structure are common. For the AI/ML community, it emphasizes that focused, domain-specific LLM adaptations can meaningfully outperform broad-capability models in key analytical and data-cleaning tasks, shaping directions for both research and practical AI deployment in data science.