We need more science in the judgment of AI (livingsystems.substack.com)

Recent discussions in the AI community highlight a concerning trend where the evaluation of Large Language Models (LLMs) is often based on subjective interpretations of impressive outputs rather than rigorous scientific criteria. The article emphasizes that without established hypotheses and evaluation metrics prior to testing, observers can easily be misled by results that appear remarkable yet lack scientific validation. This behavior, referred to as practices like "HARKing" and "p-hacking," can generate interesting hypotheses but undermines the integrity of scientific assessment. The author points out specific instances where LLMs perform well in data-rich domains, such as generating game code, but fail in areas with less training data, underscoring the importance of contextual understanding. For example, in bioinformatics, the translation of software code using LLMs has proven complex and often unreliable due to unaddressed corner cases. As a result, there's a growing call within the community to adopt a more scientific approach in LLM evaluation, moving beyond superficial judgments to ensure that claims about their capabilities are grounded in thorough testing and validation processes. This shift could enhance the reliability of AI tools and fortify their applications across diverse fields.