The same LLM is 8x slower to first token depending on who serves it (dynoyard.app)

A recent analysis revealed a stark difference in performance between two backends serving the same GLM-5.2 language model. While both backends utilized an OpenAI-compatible API, the dedicated inference host was found to be 8x faster to deliver the first token compared to a major cloud provider’s managed endpoint. Notably, the cloud endpoint, although marked as “streaming,” delivered its response in a single burst after a significant delay, essentially undermining the expected incremental response flow crucial for applications like chatbots and agents. This finding underscores the importance of thoroughly testing backend performance with realistic traffic patterns rather than relying solely on provider specifications. The results demonstrated that under typical cached conditions, the cloud provider's performance improved significantly, cutting TTFT from 10 seconds to 2.7 seconds, although it still lagged behind the dedicated host. This highlights that users must benchmark systems with true operational loads to accurately assess performance, as model behavior can vary significantly even with identical setups. The study emphasizes the need for ongoing performance evaluations, as backend configurations and serving stacks can evolve without notice, impacting the delivery experience for AI/ML applications.