No One Can Compare LLMs (xlii.space)

A recent exploration of the comparison between large language models (LLMs) reveals that evaluating which model is "better" is largely subjective and deeply personalized. The experience of users can vary significantly based on their individual working styles, communication preferences, and the context provided to the models. For instance, while one user may find Claude more effective for their needs, another might prefer ChatGPT, highlighting that what works well for one person may not be suitable for another. This dynamic is further complicated by the growing use of persistent memory in LLMs, which can yield different outputs for the same prompt based on accumulated interactions. The article suggests that traditional benchmarking tests often fail to capture this nuance, as they typically assess models in isolated environments without considering the specific styles and contexts in which developers work. Thus, it advocates for a new approach to evaluating LLMs, focusing on compatibility with individual users' prompting styles and code environments rather than strict task performance. This perspective emphasizes the importance of subjective experience in choosing LLMs, encouraging users to engage directly with models, connect them with their tools, and determine which model fits their workflow best.