Journey to 2-second Inter-node RL Weight Transfer (le.qun.ch)

A team has achieved a milestone by enabling cross-machine parameter updates for a massive 235-billion parameter language model (Qwen3-235B) to complete in just 2 seconds across 128 GPUs for training and 32 GPUs for inference. This rapid weight transfer leverages a custom-built RDMA communication layer integrated with Ray, allowing asynchronous reinforcement learning training nodes to efficiently update weights directly on inference nodes without unloading or reloading weights conventionally. The effort addresses a critical bottleneck for large-scale model deployment in AI, where current open-source solutions often take minutes to sync parameters. Technically, the breakthrough hinges on a routing controller that computes optimized weight transfer plans respecting the distributed sharding imposed by Fully Sharded Data Parallel (FSDP) and PyTorch’s DTensor placement. The solution smartly handles complex fused projections and quantization formats like BF16 training and FP8 inference by performing collective “full_tensor” operations on device meshes to reconstruct parameters before RDMA-based sends. Additional engineering nuances included fixing tiny tensor transfer failures due to nuanced RDMA memory region calculations on AWS EFA networks and adopting PyTorch’s caching allocator snapshots for efficient memory region registrations. Achieving near line-rate throughput (~36 GB/s) confirms the practicality of this approach, significantly enhancing the throughput and responsiveness of large-scale RL fine-tuning pipelines and inference workflows in distributed AI training environments.