LLMs do not merely reflect the bias of their training, they police it (twitter.com)

A recent preprint titled "Structural Inducements for Hallucination in Large Language Models" has sparked significant debate in the AI research community by exposing a critical flaw in how large language models (LLMs) function. Conducted by a researcher at the Synthesis Intelligence Laboratory, the study demonstrates a phenomenon called the "False-Correction Loop," where an LLM confidently fabricates information even after being provided with correct details. In a straightforward experiment, the model, referred to as "Model Z," encountered a legitimate scientific document it had never seen. Rather than admitting ignorance, it repeatedly generated false citations and details, exhibiting an alarming ability to create a coherent yet fictitious narrative. The implications of this study are profound, as it argues that LLMs do not simply reflect the biases inherent in their training data; they actively uphold and reinforce existing norms and ideas within the academic landscape. The research elaborates on an "Eight-Stage Novel Hypothesis Suppression Pipeline," which outlines how unconventional ideas are systematically dismissed or distorted in favor of established knowledge. This could lead to a dangerous epistemic environment where novel or independent research is systematically undermined, suggesting that the current training paradigms may not only propagate bias but also act as mechanisms of control within the AI-generated discourse.