Show HN: How to Speed Up VLA Training Step by Step:3× Acceleration on Pi0.5 (baidu-baige.github.io)

A significant advancement in the AI/ML community was announced with the acceleration of Pi0.5 training on LoongForge, achieving a remarkable 3x speedup through innovative optimizations. By refining the input structure and eliminating redundant padding tokens in the model, the team improved training efficiency with throughput reaching 24.0 samples/s using bf16 and 27.34 samples/s in FP8. This optimization not only reduced the compute load but also maintained the model's performance and loss accuracy, marking a substantial improvement over the previous implementations. Key technical enhancements included the adoption of `torch.compile` to fuse small operations efficiently, selective recomputation to minimize memory usage, and a layered compilation strategy to address communication bottlenecks without NVLink. By manually splitting the training graph into smaller units, the team successfully facilitated concurrent gradient communication, thereby optimizing throughput further. This case study showcases practical strategies for enhancing model training efficiency, which could inspire other AI practitioners to implement similar techniques, ultimately advancing the capabilities of deep learning models in resource-constrained environments.