Training a 1T-Parameter MoE on EU public infrastructure (pytorch.org)

The EUROPA consortium has successfully trained a nearly 1-trillion-parameter Mixture-of-Experts (MoE) model using the European JUPITER supercomputer, a significant milestone that challenges the belief that frontier-scale AI training necessitates exclusive, private resources. By optimizing the training process with PyTorch and TorchTitan, they improved throughput dramatically from 90 to approximately 1,300 tokens per second per GPU across a setup involving 384 NVIDIA GH200 GPUs. This achievement was made as part of the EuroHPC Frontier AI Grand Challenge, aimed at demonstrating the capability of sovereign, open-source AI within shared public infrastructure while adhering to the EU AI Act. This work is particularly crucial for the AI/ML community as it showcases how large-scale models can be trained effectively using publicly funded resources, fostering inclusivity and collaboration in AI research. The team implemented innovative strategies, such as FlexShard placement for the optimizer, activation-checkpoint offloading, and defragmenting memory streams, to enhance training efficiency while working within the unique architectural constraints of JUPITER. Their contributions to PyTorch and TorchTitan not only facilitate the scalability of future projects but also pave the way for broader access to high-capacity AI training, encouraging further investment in public compute infrastructures across Europe.