Switching an LLM's tier changes its "best tool" answer about half the time (modelsagree.com)

A recent experiment revealed that changing the tier of an AI model can significantly alter its recommendations, even when comparing models of the same brand. In a study involving various AI tools across ten categories, it was found that switching tiers would yield a different #1 recommendation roughly half the time, with only 50-65% overlap in the top-5 lists of each tier. Notably, flagship tiers often preferred established and premium tools, whereas budget tiers favored more economical alternatives. For instance, lower-tier models recommended Lambda as the best GPU cloud, while their premium counterparts chose CoreWeave. This finding is vital for the AI/ML community as it highlights the need for careful consideration of model tiers in evaluation and selection processes. Organizations monitoring AI recommendations for business decisions must recognize that the insights they receive might differ dramatically based on the tier users default to, which can influence customer perceptions and acquisition. The study underscores that relying on a single model or tier could result in potentially skewed conclusions about an AI tool's capabilities and biases, advocating for a more holistic approach in measuring AI performance and visibility across different models and tiers.