Heaven knows I'm perplexed now (idlemachines.co.uk)

Recent discussions highlight the common but often misunderstood use of perplexity as a metric for evaluating language models. While perplexity, derived from cross-entropy loss, indicates the model's uncertainty over token predictions, its raw values are frequently compared across different papers without accounting for numerous variables, rendering such comparisons largely meaningless. Instead, perplexity should be understood as an effective branching factor, reflecting the average number of choices a model faces per token. For accurate assessments, comparisons must be made with models using the same tokenizer, vocabulary, and context length, as discrepancies in these aspects can significantly skew results. Despite its limitations, perplexity serves as a valuable baseline for diagnosing issues in newly initialized models, where a perfect perplexity score should align with the vocabulary size. This principle can apply across various multiclass classification tasks, making it a useful sanity check. While perplexity remains a useful metric for tracking a model's learning process, it’s crucial for the AI/ML community to avoid conflating perplexity figures from different sources, as this can perpetuate misunderstandings within the literature.