Show HN: TurboPrefill – Multi-GPU prefill acceleration for llama.cpp (github.com)

The newly announced TurboPrefill project introduces multi-GPU prefill acceleration for the llama.cpp framework, optimizing the execution of long-context prefill workloads. By altering the scheduling strategy rather than the model itself, TurboPrefill allows multiple ubatches to enter the processing pipeline simultaneously, significantly reducing idle time between GPU stages. This approach has shown impressive results, providing speedups of up to 2.23 times compared to standard execution on long-context inputs, particularly beneficial for configurations with a higher number of GPUs. The significance of TurboPrefill lies in its ability to enhance performance for specific scenarios in the AI/ML community, particularly those utilizing layer-split mode across multiple GPUs. Its targeted optimization addresses a common bottleneck where GPUs remain idle while waiting for preceding ubatches to traverse all layers before processing the next. With potential improvements across various NVIDIA GPU architectures, TurboPrefill not only democratizes access to improved prefill throughput but also emphasizes efficient hardware utilization, paving the way for higher performance in large-scale AI deployments.