Why do OpenAI's GPT-2 weights beat mine? Part five: data quality (www.gilesthomas.com)

In a recent exploration into the performance discrepancies between OpenAI's GPT-2 models and other self-built language models, one developer delved into the significance of data quality. Despite matching the architecture of GPT-2's "small" instance—with 163 million parameters—the developer's models consistently fell short in performance during instruction fine-tuning (IFT) tasks. The study highlighted that OpenAI's models, trained on a meticulously curated dataset from high-quality web sources called "WebText," retained superior performance over competitors, even when those models were trained on ostensibly equivalent data like FineWeb, which lacked the same level of human-guided curation. The investigation emphasized that the differences in training datasets could vastly impact model quality, as OpenAI's strategies—including using weight-tying and bias adjustments—resulted in markedly better performance. The developer proposes to undertake further training runs with improved data sources, such as a curated mix of datasets that includes higher-quality educational content alongside more general data. This approach aims to discern whether better training data can enhance model performance metrics and overall effectiveness in practical applications, thus shedding light on the crucial role of data curation in training large language models.