How we made one of our largest inference workloads 4.7× more GPU-efficient (decagon.ai)

DecagonAI has achieved a remarkable 4.7× increase in GPU efficiency for one of its largest scheduled inference workloads by optimizing its serving stack rather than relying on single inference adjustments. This enhancement involved a comprehensive approach that included disaggregating the tasks of prompt processing and token generation, implementing an admission controller to better manage traffic bursts, and streamlining the time it takes for new GPU capacity to become operational. The optimization emphasizes the importance of efficiently handling bursts of requests while maximizing tokens processed per GPU-hour across all phases of the autoscaling lifecycle. Key technical improvements included designing specialized workflows for prefill and decode processes, allowing GPUs to work concurrently more effectively, thereby increasing batch sizes and overall throughput. Furthermore, an innovative queue system was integrated to ensure requests were redistributed efficiently during load spikes, minimizing bottlenecks in processing. By prefetching model files during container setup, median startup times were reduced significantly, allowing new capacity to mitigate backlogs faster. This end-to-end optimization showcases the critical need for continuous assessment and adjustment of serving mechanisms in the AI landscape, a lesson relevant for developers and engineers across the AI/ML community.