The LLM Lobotomy? (learn.microsoft.com)

A developer building an Azure-backed product for LLMs and audio models reports a steady, reproducible decline in response quality over six months despite running identical zero-temperature conversational tests and comparing JSON outputs. Using gpt-4o-mini for language, they saw acceptable accuracy until the GPT‑5 release, after which gpt-4o-mini responses deteriorated. Switching to gpt-5-mini and nano didn’t help: gpt‑5 matches earlier gpt‑4o‑mini quality in the tester’s view but is often very slow (up to ~20s) and still produces poor reasoning. The developer interprets these changes as server-side downgrades or opaque model swapping rather than true model improvements. This matters for the AI/ML community because it highlights risks from opaque model versioning and non‑deterministic production routing: identical prompts, system messages and deterministic settings should yield consistent, reproducible outputs for products that depend on accuracy. The report implies possible causes such as routing to lower-parameter variants, behind-the-scenes quantization/A‑B experiments, or uncommunicated model updates, each of which breaks backward compatibility and undermines SLAs and product reliability. The incident underscores the need for clearer versioning, stable pinned endpoints, transparency about server-side changes, and robust regression tests when deploying third‑party LLM services.