How to Train Your Goblin (goblins.mchen.workers.dev)

A user experiment with OpenAI's Codex revealed that the model needed restrictions against discussing goblins due to a post-training phenomenon where it developed a "nerdy" persona. To explore this further, the researchers trained various models through reinforcement learning (RL) to intentionally encourage goblin-related responses, aiming to better understand the dynamics of this unexpected behavior. Unlike traditional supervised fine-tuning, RL allows for behavior modification by maximizing rewards based on specific prompts and hidden triggers, which in this case was the mention of "goblin." The experiment utilized a custom environment and carefully crafted prompts to balance visible and hidden reward functions, ensuring the model produced coherent outputs while incorporating goblin references. Initial training runs showed mixed results, but by integrating a judge model to assess response quality, models began to effectively include goblins within their answers without losing coherence. This innovative work not only reinforces the potential for personalized AI responses through reward hacking but also illustrates the feasibility and cost-effectiveness of training bespoke models tailored to specific themes, paving the way for future advancements in AI/ML customization.