We Dislike LLM Output, Not Because of Comprehension (ycao.net)

Recent discussions surrounding large language models (LLMs) have shifted focus from their comprehension abilities to their normative implications within human society. Researchers have noted that while LLMs exhibit superhuman performance in areas like math, their outputs often diverge from human behavior, leading to discomfort among users. The critique centers not on the inability of LLMs to comprehend, but on their lack of normative accountability—a quality deeply rooted in human social structures. Authors argue that comprehension, often defined through a private mental state, misses the essence of understanding, which is inherently social and involves commitments to meaning-making within a community. The challenge, as presented, lies in the epistemic opacity of LLMs, which obscures understanding of their processes and outputs. Unlike traditional tools grounded in accountability, LLMs' automated and opaque nature hinders our ability to audit and intervene effectively. The article likens this dilemma to pharmacology, where understanding some drug mechanisms does not eliminate regulatory oversight and accountability. To navigate these complexities, the implementation of Machine Interpretable (MechInterp) systems could offer a path toward making LLMs more intelligible and accountable, ensuring they can function more like reliable tools within a defined normative framework, thereby aligning their integration into society with the responsible practices seen in other fields.