Model is being retired. Are you ready to upgrade? (mahalsystems.ai)

OpenAI recently retired five fine-tuned models and several legacy GPT snapshots, coinciding with Anthropic's deprecation of Claude Haiku 4.5. This shift compels users to upgrade, prompting questions about the implications of such migrations. While both companies suggest easy model swaps that could improve performance, tests show that upgrades do not guarantee better results, with some models performing worse in specific tasks despite overall better averages. Notably, at least one model downgrade occurred in 3 out of 12 comparisons, highlighting the risks inherent in automatic upgrades. Significantly, the findings reveal that merely switching to a lab's recommended successor may not yield the best outcomes. Cross-testing models from different labs frequently outperformed the suggested successors, suggesting that organizations should evaluate alternatives rather than sticking with one provider. Furthermore, migrating to a new model often necessitates re-tuning of prompts and configurations, which complicates the upgrade process. The takeaway for the AI/ML community is to approach model upgrades as an opportunity for rigorous evaluation and optimization, reinforcing the need for ongoing assessments of model performance rather than relying solely on the latest releases.