GPT-2 124M checkpoint pre-trained on OpenWebText 27.5B tokens (github.com)

The recent release of a 124M-parameter GPT-2 model, trained from scratch on the OpenWebText dataset, showcases a significant achievement in the AI community. This model, built using a custom deep learning library, achieved a validation loss of 2.764 and a perplexity score of 15.87 after 56,000 training steps out of a planned 600,000. The model was exposed to approximately 27.5 billion tokens, highlighting its substantial training data. This initiative demonstrates the potential for developing competitive baseline models without relying on popular frameworks like PyTorch, illustrating alternatives in the deep learning landscape. The significance of this development lies in its emphasis on educational value by illustrating how a bespoke deep learning library can effectively train a complex model like GPT-2. Despite being undertrained based on its full schedule, the model's performance metrics, including bits-per-byte and various perplexity scores, indicate that it closely approaches the capabilities of existing models from Hugging Face. This experiment not only sheds light on the intricacies of training generative models but also highlights the versatility of different architectures and training methodologies within the AI/ML community.