RL economics, morally charged terms, and "distillation" (thomasdullien.github.io)

A recent discussion in the AI community has highlighted the evolving dynamics of training large language models (LLMs) using reinforcement learning (RL) and the implications of using particular terminology, such as "distillation." The conversation emphasizes that model training now relies heavily on generating solutions through RL because traditional human-curated data sources have become scarce. RL allows models to produce more effective solutions by testing multiple rollouts and using successful outcomes to refine their training, but this process is computationally expensive and creates a performance plateau. The implications for the AI sector are significant: while early developers invest heavily in creating robust models, subsequent entrants can leverage these outputs, benefiting from cheaper data generation. Furthermore, there's a contentious debate surrounding copyright and the ethical framing of business models in AI, specifically how companies perceive others training on their models' output. Some companies have coined the term "distillation attacks" to cast this practice as morally dubious, while critics argue that it merely serves to protect their business interests. The article argues for clearer terminology, suggesting "training on model output" instead of "distillation," to avoid conflating technical processes with moral judgments. This perspective calls for a more honest discourse about copyright issues and the financial motivations that often drive moral framing within the rapidly evolving AI landscape.