AutoSP: Long-Context LLM Training via Compiler-Based Sequence Parallelism (pytorch.org)

AutoSP has been introduced as a game-changing solution for training large language models (LLMs) with extremely long contexts exceeding 100,000 tokens. Traditional methods often lead to out-of-memory (OOM) issues, even when employing advanced training techniques like ZeRO/FSDP. AutoSP simplifies this process by using a fully automated compiler-based approach to convert standard training code into multi-GPU sequence parallel code, enabling researchers to efficiently manage longer input contexts without the heavy code modifications previously required. This innovation is significant for the AI/ML community as it democratizes access to long-context training capabilities, allowing users to seamlessly integrate AutoSP into existing workflows via DeepSpeed with minimal effort. Key technical features include its compatibility with various parallel training strategies, like ZeRO, and an efficient custom activation-checkpointing strategy tailored for long contexts. Initial evaluations show that AutoSP not only increases the maximum trainable sequence length but does so with negligible impact on runtime performance, making it a potent tool for researchers aiming to push the boundaries of LLM capabilities.