Piper: A Programmable Distributed Training System (syfi.cs.washington.edu)

Piper, a new programmable distributed training system for PyTorch, has been introduced to streamline the creation of complex training schedules without the need for specialized runtimes. As large-scale machine learning models increasingly utilize multiple parallelism strategies—like pipeline, data, and expert parallelism—Piper addresses the limitations of existing frameworks such as Megatron and DeepSpeed by allowing users to decouple model placement and GPU scheduling from the runtime. This flexibility enables enhanced control over scheduling choices, accommodating various model architectures and communication patterns, which is crucial for optimizing performance. The system works by allowing users to annotate their model code with lightweight tags that identify schedulable regions, facilitating the application of scheduling directives through its compiler. Piper generates a global training directed acyclic graph (DAG) that encodes all necessary compute, communication, and placement details, which is then decomposed into execution plans for individual devices. This architecture simplifies the execution of complex training strategies, such as the DualPipe schedule, which overlaps computations with communication operations. By enhancing user control and reducing communication overhead, Piper promises significant advancements in the efficiency of training large models, potentially benefiting a broad spectrum of AI/ML applications.