67% Cost Savings with PD Disaggregation Using Ray and vLLM on AMD MI325X (www.anyscale.com)

Recent advancements in AI/ML infrastructure have led to significant cost savings using Prefill-Decode (PD) disaggregation with Ray and vLLM on the AMD MI325X GPU. This approach optimizes the serving of large language models (LLMs) by isolating the prefill and decode phases onto dedicated hardware, achieving up to 67% reduction in compute costs and a 2.7x improvement in goodput (queries per second). By avoiding competition for resources between these phases, the system can handle more requests while maintaining latency Service Level Agreements (SLAs). However, this method introduces operational complexities, particularly around the transfer of key-value (KV) caches across nodes, which can affect responsiveness. While PD disaggregation enhances throughput for longer outputs, it can be slower for time-to-first-token (TTFT) metrics, making it unsuitable for workloads where immediate responsiveness is crucial. The need to optimize the prefill-to-decode (P:D) ratio according to specific workloads is highlighted, as improper ratio settings can negate the benefits of PD. Insights from extensive testing guide practitioners in determining ideal configurations for maximizing performance depending on their operational needs.