Finetuning a Reasoning LLM with Supervised or Reinforcement Learning? (discuss.huggingface.co)

Recent insights into fine-tuning reasoning large language models (LLMs) emphasize the importance of structured training approaches, such as Supervised Fine-Tuning (SFT) and Preference Optimization (DPO). The discussion outlines practical strategies for developing effective training datasets, indicating that SFT is the ideal starting point when high-quality demonstration data is available. It advocates for detailed attention to how training data is represented, particularly separating raw annotations from model-specific formats, to ensure the correct behaviors are learned. Significantly, the article stresses that merely fine-tuning on tool-call examples isn't sufficient; models must be exposed to various scenarios, such as “no-tool” situations and clarification requests, to avoid over-relying on tool calls. Additionally, the paper suggests that DPO can serve as a beneficial intermediate step before employing Reinforcement Learning (RL), allowing for the optimization of model outputs through preferred trajectory pairs. This layered approach not only enhances the model's immediate performance but also lays the groundwork for more complex learning through RL when executable environments and reliable reward systems are established. Overall, these findings aim to refine the training methodologies for LLMs, ultimately fostering more capable and context-aware AI systems.