We brought ViT-H coordination below a 16 GB memory budget (nodeox.com)

A recent engineering breakthrough has successfully reduced the memory usage for the ViT-H coordination process below a 16 GB budget, dramatically down from nearly 30 GB during full runs. The optimization involved innovative changes in how the model state was managed, particularly in eliminating unnecessary memory usage by identifying and breaking reference cycles within the aggregation code. After refining their process through multiple iterations, engineers were able to stabilize memory requirements and ensure that working memory was efficiently released post-training. This advancement is significant for the AI/ML community as it enhances the efficiency of model training, particularly in distributed settings. The ability to conduct fine-tuning with lower memory overhead not only allows for better resource allocation across multiple GPU hosts but also improves overall system performance. The experiments indicated a balance between memory use and execution time, demonstrating a trade-off of about 1.5 seconds per round in exchange for a substantial reduction of around 20 GB in peak memory. This development serves as a critical step towards optimizing large model training workflows, making it more feasible for researchers and practitioners to utilize advanced models like ViT-H in real-world applications.