Pulse: Decentralized RL training centralized speed (100x weight sync reduction) (arxiv.org)

Researchers have introduced PULSE (Patch Updates via Lossless Sparse Encoding), a novel method to address the significant challenges of weight synchronization in decentralized reinforcement learning (RL). This breakthrough is particularly crucial as RL becomes increasingly integral to fine-tuning large language models (LLMs) in distributed learning environments, where bandwidth limitations often hinder scalability. The study reveals that over 99% of model parameters remain unchanged during updates, supporting the assertion that only sparse updates are necessary for effective training. PULSE leverages this update sparsity by efficiently transmitting only the indices and values of modified parameters, resulting in a dramatic 100x reduction in communication volume—from 14 GB to approximately 108 MB—while ensuring bit-identical training dynamics to traditional methods. This optimization allows for compressed weight synchronization even in low-bandwidth settings, drastically lowering the required bandwidth from 20 Gbit/s to just 0.2 Gbit/s, thereby enhancing GPU utilization during decentralized training. As standard RL frameworks struggle with communication overhead, PULSE represents a significant advancement, paving the way for more efficient distributed training of complex AI models.