A Bitter Lesson for Data Filtering (arxiv.org)

Recent research into data filtering for large model pretraining challenges the conventional wisdom that high-quality data is crucial for model performance. The study reveals that when large parameter models are sufficiently trained with ample computational resources, they can actually thrive on low-quality or "distractor" data. This suggests that the best filtering approach may well be to use all available data rather than curating it strictly for quality. This finding has significant implications for the AI/ML community, particularly in developing and training large-scale models. It opens up new avenues for leveraging vast datasets that may previously have been dismissed as inadequate. By demonstrating that models can benefit from diverse data inputs, researchers may reconsider their data curation strategies, potentially reducing the time and effort spent on filtering while also expanding the accessibility of large datasets for model training.