TurboPrefill: 3.27× Prefill Speedup in Llama.cpp (devpost.com)

A new advancement in large language model efficiency has emerged with the announcement of TurboPrefill by Trykhlieb, which addresses significant communication bottlenecks between GPUs during the prefill phase. Traditionally, large models utilizing multiple GPUs face limitations not only in computational power but also in the data transfer speeds, particularly in systems lacking high-speed interconnects like NVLink. TurboPrefill optimizes this by rearranging the scheduling of prompt microbatches, allowing several GPUs to work concurrently rather than sequentially. This pipelining method enhances prefill performance, particularly in configurations where inter-GPU communication is a constraint. During the OpenAI Build Week, an experimental port of TurboPrefill was developed for ik_llama.cpp, demonstrating notable performance improvements across various context sizes. For example, the prefill speed for the GPT-OSS-20B model increased substantially from 184 to 602 tokens per second at a context length of 1,024 tokens. The project's reliance on tools like Codex and GPT-5.6 highlighted the potential for AI to expedite low-level systems work, although the main challenge lay in maintaining correct execution amid the complex changes made to microbatch scheduling. Overall, this innovation showcases the significant role of optimized scheduling in enhancing AI model performance, particularly in bandwidth-limited environments.