TurboPrefill: 2.7× faster than llama.cpp Pipeline Parallel on Llama-3-70B (github.com)

Researchers have announced TurboPrefill, a novel implementation that enhances the efficiency of multi-GPU prefill processes by leveraging Intra-Prompt Pipeline Scheduling. This innovative approach significantly reduces the waiting time for answer generation from 9.1 seconds to just 4.6 seconds, demonstrating a remarkable 2.7× speed increase over the llama.cpp Pipeline Parallel framework when applied to the Llama-3-70B model. This makes TurboPrefill especially valuable for vision language models (VLMs) where rapid response times are critical. TurboPrefill addresses limitations in data transfer between GPUs, allowing it to excel even in configurations with constrained bandwidth. For instance, in scenarios where GPUs are connected through slower interfaces, TurboPrefill can achieve up to 5× speedup compared to existing methods like Pipeline Parallel and SM Tensor. As a result, not only does TurboPrefill facilitate improved memory scaling across multiple GPUs, but it also enhances overall throughput for data-intensive tasks. This advancement represents a significant step forward in optimizing AI model performance, especially for applications requiring real-time processing.