Speedrunning an RL Environment (sidb.in)

A contributor to Prime Intellect’s Environment Hub published a practical “speedrun” on building RL environments for LLMs using the verifiers framework and demonstrates adapting the AgentDojo benchmark. The post frames RL environments as programmable “hamster mazes” where rollouts (sequences of states, actions, rewards) let you evaluate or train agents. Verifiers is pitched as a reusable harness that standardizes dataset formats, multi-turn interactions, tool usage, reward calculation and resource setup so you can convert existing benchmarks into RL-compatible evaluations without rewriting bespoke scaffolding. The author points to the prime CLI for boilerplate generation and recommends adapting upstream frameworks rather than patching locally. Technically, verifiers exposes base env classes (vf.SingleTurnEnv, vf.MultiTurnEnv with env_response and is_completed hooks, vf.ToolEnv with add_tool/call_tool, vf.StatefulToolEnv with update_tool_args, and vf.MCPEnv) and a clear lifecycle: create_dataset → __init__ → load_environment → per-task init_state/setup_state → conversation loop (LLM calls, env_response, tool calls) → rubric.score for reward outputs. The AgentDojo case highlights why this matters: tasks include “user” and “attacker/injection” suites that surface prompt-injection vulnerabilities across three scenarios (direct, indirect/injected, injection-only). With current LLMs completing ~66% of tasks unperturbed, the framework’s primitives make it easier to benchmark robustness, automate tool interactions, and integrate security testing into RL training/evaluation workflows.