Pipeline-parallel LLM inference across GPUs on separate machines (github.com)

A new groundbreaking approach to pipeline-parallel inference for large language models (LLMs) has been announced, enabling a 744-billion-parameter model to be distributed across GPUs located on different machines and networks. This innovative method, implemented by the Shard framework, divides the model into contiguous blocks, allowing each GPU to handle only a segment of the entire architecture. By streaming activations through these segments, the system can achieve an output rate of around 30 tokens per second, a significant advancement given that previous attempts at similar setups faced substantial latency issues. The importance of this development for the AI/ML community lies in its potential to dramatically enhance the scalability and accessibility of large models. Unlike traditional methods where a single machine or data center held the entire model, this decentralized approach minimizes bottlenecks by leveraging distributed resources. Key technical innovations include speculative decoding, which allows multiple tokens to be processed simultaneously, and CUDA graphing, which optimizes the decoding pipeline, reducing overhead. As a result, this framework not only addresses the challenges of latency but also paves the way for future developments in permissionless swarm infrastructures, enabling anyone to contribute computing resources to run large models without centralized control.