Science or Slop? (yerimoh.github.io)

A new evaluation tool known as the Science Slop Index has been introduced to scrutinize scientific papers, particularly those generated or revised by AI. Unlike typical token-level AI detectors that focus on sentence predictability, the Science Slop Index analyzes the coherence of scientific reasoning within papers. It assesses how well different sections of a paper connect, evaluates the arguments and citations presented, and inspects the methodologies and evidence shared. This tool provides a unique approach to identifying "slop" in scientific writing, effectively measuring the structural integrity of scientific discourse. The significance of this development for the AI/ML community lies in its ability to differentiate between human-written and AI-generated research based on the quality of scientific argumentation rather than mere linguistic accuracy. The index employs six measures across three evaluative planes, resulting in an overall score that reflects the level of scientific coherence, with higher scores indicating greater alignment with traditional scientific standards. This could help uphold research integrity amidst the growing production of AI-driven content, ensuring that scientific advancements are based on robust reasoning and well-substantiated claims. Researchers can also contribute to its development by proposing new patterns of scientific slop for consideration.