Open-Weight Masked Introspection: Measuring What LMs Can Report About Their Own (arxiv.org)

Researchers have introduced a framework called Open-Weight Masked Introspection (OWMI) to investigate whether language models (LMs) can accurately introspect on their own computations. By examining eight open-weight models from seven families, the study aimed to determine if these models could audit their internal states and discern changes made during processing. However, the tests concluded that none of the models performed better than chance when asked to identify alterations, with an AUROC score close to 0.5007. This highlights a critical limitation: while models possess the necessary information, the pathway from internal state understanding to verbal reporting remains flawed. The findings have significant implications for the AI/ML community, particularly regarding the reliability of models in self-assessment and introspective capabilities. While current open-weight models struggled with introspection, the research indicates that future developments may still yield better results with advancements in model architecture and training methodologies. Furthermore, the study emphasizes the need for rigorous validation methods to accurately interpret model outputs, strengthening the foundation for oversight in applications leveraging AI introspection.