Self play and autocurricula in the age of agents (www.amplifypartners.com)

Researchers are revisiting self-play and autocurricula — the automatic progression of tasks that drove AlphaGo — as a way to train modern LLM-based agents. Recent academic efforts (Natasha Jaques’ Spiral, Andrew Zhao’s AbsoluteZero) and industry-adjacent work (Kimi‑k1.5’s curriculum sampling) show early wins: AbsoluteZero jointly optimizes a single model as both teacher and student to generate and solve math/code problems, rewarding the teacher for producing “learnable” (not impossibly hard) tasks; Kimi‑k1.5 used prioritized sampling to boost accuracy by ~5 points; LILO and SPICE replace naive difficulty metrics with variance-based learnability measures and report faster convergence and broader reasoning gains. Still, experts like Noam Brown caution that self-play is straightforward in two‑player zero‑sum games but much harder in open-ended real‑world domains. Technically, the key idea is shifting from fixed, human-designed datasets to a dynamic loop where a generator (teacher) proposes tasks at the model’s learnable frontier and a learner (student) solves them, maximizing information gain and modular skill composition. That raises crucial challenges: designing teacher rewards to avoid trivial or adversarial task hacks (e.g., coin‑flip tasks that fake a target success rate), ensuring stability under joint optimization, and scaling unsupervised environment design. If solved, autocurricula could automate intermediate-task generation, dramatically improve sample efficiency, domain adaptation and creativity in foundation models — but robust reward objectives and defenses against reward-hacking remain the bottleneck.