Reducing Pipeline Bubbles with Adaptive Parallelism on Heterogeneous Models (arxiv.org)

Researchers introduced AdaPtis, a new LLM training system that reduces pipeline bubbles by applying adaptive pipeline parallelism tailored to heterogeneous model architectures. Pipeline bubbles—idle gaps in staged execution—grow worse as models become more varied across layers, throttling throughput. AdaPtis addresses this by building a pipeline performance model that accurately predicts training throughput, then jointly optimizing model partitioning, placement across devices, and workload scheduling using that model. A unified pipeline executor supports a range of pipeline strategies so the optimized plans can be executed efficiently in practice. For the AI/ML community, AdaPtis demonstrates that co-optimizing partition, placement and scheduling—rather than tuning them independently—can substantially improve end-to-end training efficiency for diverse LLMs. In experiments across multiple architectures and scales, AdaPtis speeds training by an average of 1.42× and up to 2.14× versus a Megatron-LM I-1F1B baseline. The work highlights the value of accurate performance modeling and adaptive execution layers for improving resource utilization, lowering training time and cost, and making pipeline parallelism robust to architectural heterogeneity in modern LLMs.