Pimmur, can LLM simulate human collective behavior? (arxiv.org)

Recent research has introduced the PIMMUR principles, shedding light on the methodological shortcomings in the use of large language models (LLMs) to simulate human collective behavior. An audit of 39 studies revealed that 89.7% breached at least one of the PIMMUR principles—pertaining to agent profiles, interactions, memory, control, unawareness, and realism—raising serious concerns about the validity of these "AI societies." The analysis found that frontier LLMs only identified the underlying social experiment in 50.8% of cases, indicating a lack of understanding, while excessive control in 61.0% of prompts skewed outcomes significantly. The findings are pivotal for the AI/ML community, suggesting that many previously reported collective phenomena may be artifacts of flawed methodologies rather than genuine social dynamics. By reproducing various social experiments, the researchers showed that enforcing the PIMMUR principles often nullifies or alters the emergence of behaviors attributed to these AI-driven simulations. This raises critical implications regarding the reliability of LLMs as tools for scientific exploration of human society, suggesting that they may reflect model-specific biases instead of capturing universal human behaviors, thereby highlighting the need for more rigorous standards in simulating complex social interactions.