RL Doesn't Work on Slurm (blog.skypilot.co)

Recent discussions in the AI community highlight significant challenges with using Slurm, a popular batch scheduler, for online reinforcement learning (RL). Various frameworks, including OpenRLHF and veRL, are struggling due to Slurm's inability to effectively orchestrate multi-service architectures essential for online RL training. Unlike traditional batch jobs, online RL requires continuous interactions between multiple processes, each needing to share resources dynamically and can often run indefinitely. This dependency on continuous service and real-time communication between grouped processes exposes Slurm's core limitations, leading to inefficient workarounds and overhead that impacts model training and performance. The failure of Slurm to adapt to the needs of modern online RL workflows has prompted organizations to explore alternatives like Kubernetes. Companies like Meta and H Company have migrated their infrastructures to more dynamic platforms that allow for service discovery and health checks at a granular level, enhancing task management. The emergence of tools like SkyPilot Job Groups illustrates a move towards flexible orchestration options that can manage complex ML tasks as cohesive units. This shift signals a growing recognition within the AI community that traditional schedulers may need to evolve or be replaced to facilitate the next generation of AI research.