How do LLMs understand an artist's inner universe? (jimiwen.substack.com)

Researchers have developed a novel evaluation methodology called "How Does It Taste?" to assess how well language models (LLMs) can interpret complex artistic texts without prior context. This test challenges LLMs by presenting them with a singular, unfamiliar scripture, 《思無字》, and asking them to derive meaning solely from the text itself, rather than relying on external information or frameworks. The aim is to determine if the models can genuinely "taste" the work, which entails a nuanced understanding that goes beyond basic theme summarization to engage with the text's deeper aesthetic and metaphysical dimensions. This approach is significant for the advancement of AI and machine learning as it pushes the boundaries of what LLMs can achieve in literary interpretation. The test measures various competencies, including the model's ability to maintain an "immanent sensibility," analyze literary features, and explore the text's ontological implications. The scoring system, currently at version 8, emphasizes the importance of a model's capability to immerse itself in the work and experience its nuances without collapsing into superficial readings. By delineating how well models can navigate complex artistic judgments, this research may pave the way for greater emotional and critical engagement in AI interpretations, potentially transforming how we conceive LLMs' roles in creative and analytical domains.