Torchforge – a PyTorch native library for scalable RL post-training (pytorch.org)

Meta announced torchforge, a PyTorch-native library that abstracts the infrastructure of large-scale reinforcement learning so researchers can focus on algorithms. Torchforge exposes a simple, composable API that lets you write rollout logic as readable pseudocode (e.g., a single generate_episode function) and then compose it into synchronous on‑policy loops or fully asynchronous off‑policy systems without changing RL code. It targets LLM use cases where policies are big, sharded models and rewards often require costly autoregressive inference or external verification (tool use, test execution), addressing real bottlenecks like stale-policy rollouts, replay buffers, and painfully slow weight broadcasts across many replicas. Technically, torchforge builds on Monarch (an actor/mesh-based PyTorch distributed controller) to hide SPMD complexity and orchestrate sharded components (vLLM for inference, torchtitan/FSDP for training) via ActorMeshes, RDMA transfers, and coordinated fanout updates. That design enables any degree of asynchrony, fault tolerance, and scaling of heterogeneous workers while preserving high throughput. Torchforge is experimental and evolving, but it promises to accelerate RL research for LLMs by replacing ad‑hoc infra engineering (resharding, synchronization, retries) with a single-controller programming model that composes proven production pieces.