The Empiricism Gap in Computer Science (doomscrollingbabel.manoel.xyz)

A recent discussion highlights the "Empiricism Gap" in computer science, specifically critiquing two dominant methodologies: "build-and-test" and "describe-and-defend" empiricism. The former is prevalent in machine learning (ML) and involves creating models and systems, evaluating their performance via shared benchmarks like the ImageNet competition. This iterative, artifact-driven approach accelerates research but risks overfitting and losing sight of broader claims. For example, many in ML have continued to produce research based on established benchmarks without revisiting the validity of prior results, as exemplified by the community's response to challenges posed by new findings on ImageNet’s effectiveness. In contrast, "describe-and-defend" empiricism, often seen in disciplines like economics and political science, prioritizes formulating hypotheses and justifying claims with robust evidence. This method encourages rigorous scrutiny and robustness checks, promoting a culture of continuous reassessment of empirical findings. The critique suggests that computer science, particularly in areas lacking clear benchmarks, often defaults to a build-and-test mentality, producing research that appears empirical yet fails to provide generalizable insights. Addressing this gap is vital for enhancing the credibility of computer science research and its impact on real-world applications, especially in high-stakes areas like human-computer interaction and policy development.