Misconduct in Post-Selections and Deep Learning (2024) (arxiv.org)

A recent theoretical paper titled "Misconduct in Post-Selections and Deep Learning" explores critical issues of data integrity in the machine learning field. The author argues that most deep learning methodologies, irrespective of the learning mode (supervised, reinforcement, etc.), are plagued by two main forms of misconduct: "cheating in the absence of a test" and "hiding bad-looking data." The paper emphasizes that authors should disclose comprehensive validation results, including the average errors of all trained models, to combat these issues. Significantly, the paper critiques the common practice of using cross-validation to mitigate the effects of post-selection bias in model evaluation, asserting that it does not provide a valid defense against the identified misconducts. This finding calls into question the reliability of current statistical practices in machine learning, prompting a reevaluation of how model performance is reported and validated. By highlighting these fundamental flaws in evaluation methods, the paper contributes to an ongoing discourse about transparency and integrity in AI research, a crucial concern for maintaining trust and accountability in the rapidly evolving AI/ML community.