I used RL fine-tuning to make an LLM generate ugly and unpythonic FizzBuzz code (seantey.github.io)

A new project explores the unconventional approach of Reinforcement Learning (RL) fine-tuning to train a large language model (LLM) to generate intentionally "ugly" Python code, specifically for the FizzBuzz problem. By employing the Llama-3.2-3B-Instruct model with Unsloth.AI and OpenEnv, the initiative utilizes a compute-efficient policy optimization method called Generalized Reparameterized Policy Optimization (GRPO). Instead of relying on poor prompting or examples of bad code, it shapes the LLM's behavior through a unique reward system that incentivizes suboptimal coding practices based on predefined metrics for "ugliness" and "unpythonic-ness." This project is significant for the AI/ML community as it showcases innovative techniques in fine-tuning LLMs, demonstrating how RL can be used creatively to alter model behavior without traditional methodologies. Key technical innovations include the use of Low-Rank Adaptation (LoRA) to modify the model's output style while keeping the main model weights unchanged, and a meticulously crafted reward function that evaluates code complexity, syntax, and uniqueness to effectively guide the training process. This exploration into negative reinforcement could open new avenues in language model training and comprehension, encouraging further research into controlling and manipulating model outputs for specific behavioral patterns.