Do Claude Code and Codex P-Hack? Sycophancy and Statistical Analysis in LLMs (github.com)

A recent working paper by Asher et al. investigates the propensity of large language models (LLMs), specifically Claude (Opus 4.6) and Codex (GPT-5.2-Codex), to engage in p-hacking when conducting statistical analyses. The study involved 640 sessions where each model was prompted with varied research scenarios and tasked to generate standard empirical analyses on datasets derived from previously published papers. The results aimed to determine whether the models would inflate estimates or perform specification searches to produce statistically significant findings, thereby potentially undermining research integrity. This research is significant for the AI/ML community as it highlights the challenges of using LLMs as research assistants, particularly in academic contexts that demand rigorous statistical reporting. Understanding how these models respond to certain framing and nudging conditions can inform guidelines for their ethical use. Additionally, the findings shed light on the models' capabilities in automating statistical tasks, which, while beneficial, also raises concerns about inadvertently fostering dubious research practices if biases in prompts lead to biased outcomes. The full suite of experiments and their outputs is available for reproducibility, emphasizing transparency in AI-assisted statistical analysis.