I trained a 30M-param LLM from scratch and the scaling "floor" was a mirage (github.com)

A developer has successfully trained a 30 million parameter decoder-only transformer model from scratch as part of a comprehensive learning project, utilizing Kaggle's free T4 GPUs and the TinyStories dataset. The purpose of this project was to delve deeply into the intricacies of transformers by manually implementing key components, including the model architecture, training loop, data pipeline, and tokenizer. The project culminated in a technical write-up outlining various controlled experiments, including ablation studies that challenged the conventional wisdom surrounding model scaling and performance. This project is significant for the AI/ML community as it sheds light on the nuanced relationship between model size, data, and compute resources, suggesting that the observed "scaling floor" may not be as definitive as previously thought. Through a series of experiments, the author demonstrated how careful hyperparameter tuning and alternative normalization techniques, such as RMSNorm versus LayerNorm, can significantly influence training outcomes. Ultimately, the model produced coherent narrative text that showcases its capabilities despite being smaller in scale, highlighting the potential for specialization in model design to outperform sheer scale in specific applications. This endeavor serves as a valuable resource for researchers and practitioners interested in building and optimizing their own language models.