Making post-publication code checks a first-class research artifact (www.arxiv.org)

A new position paper advocates for the implementation of standardized post-publication code verification in academic journals, particularly within the rapidly evolving field of machine learning. Despite increasing demands for code and data availability, reproducibility remains a significant challenge due to factors like the reliance on specialized hardware and the dominance of conference publications that limit thorough peer review. The paper proposes a system where independent researchers can submit code replications after publication, earning verification badges that would be displayed in the article metadata. This approach shifts some verification responsibility from overwhelmed reviewers to the research community. The significance of this initiative lies in its potential to enhance transparency and trust in machine learning research by formalizing post-publication verification processes. By recognizing independent replications through badges, the initiative hopes to foster a more collaborative environment that encourages reproducibility. Furthermore, the authors stress that while journal policies have begun to address reproducibility at the pre-publication stage, no formal post-publication verification mechanisms currently exist. Implementing such frameworks could address the limitations of traditional review processes and ultimately lead to more reliable findings in AI and machine learning research.