Predict your distributed LLM training time before you burn GPU hours (github.com)

A new tool designed for predicting training times of large language models (LLMs) across multiple GPUs has been announced, allowing users to estimate wall-clock time before initiating training. By leveraging advanced 3D parallelism techniques, including pipeline, tensor, and data parallelism, this tool aims to help researchers effectively plan their computational resources and evaluate different parallelization strategies without incurring costly trial runs. The package will soon be available on PyPI, but can currently be installed directly from its GitHub repository. The significance of this tool lies in its ability to run predictions on CPU using pre-trained regressors for popular GPU models like NVIDIA A100 and GH200, making it accessible for initial usage even without a GPU. Users can input various configurations and receive precise training time estimates, which can aid in optimizing workflows and reducing wasted computational hours. The project's backing by the National Science Foundation illustrates its importance in advancing the efficiency of distributed deep learning and offers extensive customization options for researchers looking to refine their training setups.