Miles v0.1: Production-level Post-training (www.lmsys.org)

Miles v0.1 has been released as a robust, production-ready system designed for frontier post-training in reinforcement learning (RL). This update builds on the previous Miles iteration and emphasizes verified, clean, and customizable optimization across the RL training loop. Key features include a fully asynchronous RL process that ensures uninterrupted rollout generation and model training, which enhances efficiency and scalability. The integration with SGLang allows for fast, multi-turn agentic rollouts, maintaining a high cache-hit rate and ensuring that training does not lag due to slow trajectories. This development is significant for the AI/ML community as it simplifies the implementation of advanced RL workloads, making them more accessible to researchers and developers. Miles' capabilities include a Token-In-Token-Out (TITO) system that preserves token-level precision during training and an efficient Rollout Routing Replay (R3) mechanism that mitigates numerical discrepancies between rollout and training phases. Furthermore, it supports various low-precision training configurations and intricate memory optimizations, making it viable to train large-scale models efficiently. These features collectively position Miles v0.1 as a powerful tool in the ongoing advancement of reinforcement learning technologies.