Digging into the LLM-as-a-Judge Results (www.gilesthomas.com)

In the ongoing exploration of training large language models (LLMs), a new analysis reveals inconsistencies in the evaluation of various LLMs using a structure called LLM-as-a-judge. The examination is part of creator Sebastian Raschka's ongoing projects after detailing the LLM-building process. Four models, trained on different configurations, were assessed against OpenAI’s original GPT-2 weights, utilizing a method that compares cross-entropy loss to instruction fine-tuning (IFT) scores. The results highlight a puzzling lack of correlation between loss and IFT scores, prompting further investigation into the assessment methodology. This investigation is crucial for the AI/ML community as it underscores the complexities of model evaluation and the importance of comparative analysis in fine-tuning LLMs. The results suggest that both the predictive capabilities (loss) and the quality of the training data (knowledgeable vs. smart models) are significant factors influencing performance. The author proposes a batch scoring method to ensure consistency in evaluations, which aims to refine model comparisons and enhance the interpretation of model performance. This insight calls for a deeper understanding of the training data's impact on model intelligence and instruction-following ability, offering valuable implications for future research and development in the field.