Toward RL on mixed-precision, mixed-accelerator samplers (modal.com)

A recent study has introduced a novel approach to reinforcement learning (RL) that employs mixed-precision and mixed-accelerator samplers, challenging the traditional practice of using uniform accelerators and precision levels for both samplers and trainers. By allowing samplers to operate on different accelerators and using lower precision formats such as FP8, the study reveals a significant opportunity to optimize resource usage since samplers consume 80-85% of GPU resources during large RL runs. The research demonstrates that off-policy correction techniques can effectively stabilize training despite the heightened mismatch typically introduced by using mixed-precision and different hardware. The implications of this research are profound for the AI/ML community, as it paves the way for more cost-effective and efficient RL training without sacrificing performance. The experiment utilized a Qwen3.6-35B-A3B model on coding tasks, revealing that the mixed samplers could achieve performance comparable to homogeneous samplers while reducing costs by about 15%. This innovative framework, coupled with asynchronous training and specific correction techniques such as masked importance sampling, offers a flexible and scalable method for enhancing RL applications, making this work a significant advancement in the field.