Is more training data always better? (blog.oumi.ai)

Researchers revisiting fine-tuning ask whether "more data is always better" and find a surprising answer: careful curation can beat scale. The LIMA study fine-tuned a Llama-65B base model on a 1,000-sample dataset (mix of community forums, Stack Exchange, WikiHow, Reddit and hand-written examples), training all weights for ~15 epochs and selecting a checkpoint via held-out evaluation. Evaluated on 300 challenging prompts with human and GPT-4 judges, the tiny fine-tuned model produced responses judged equivalent or preferred to GPT-4 about 43% of the time and even outperformed the same base model fine-tuned on 52k samples. Ablations show diversity of examples matters far more than sheer quantity. Technically, LIMA’s results highlight that foundation models store broad capabilities from massive pretraining and that fine-tuning often re-weights or selects subdistributions of those abilities (the paper’s "Superficial Alignment Hypothesis"). The experiment used full-weight supervised fine-tuning (no LoRA, RLHF, or DPO), suggesting careful dataset design can substitute for scale—but also that practitioners should match model capacity to data size and consider parameter-efficient methods (LoRA) for tiny datasets. Caveats include reliance on a 65B base model and specially curated data; subsequent work (e.g., LIMO) reports similar trends, indicating this "less-is-more" effect is worth testing across tasks and deployment settings.