Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch (github.com)

A new project named NanoEuler has emerged, featuring a GPT-2-scale language model built entirely from scratch using C/CUDA, without relying on popular machine learning libraries like PyTorch. This endeavor showcases a comprehensive training pipeline, including a custom byte-level BPE tokenizer, pretraining on a mixture of books and web data, and plans for supervised fine-tuning into a chat model. The model, which has approximately 116 million parameters, can train on a single RTX 4070 GPU and includes meticulous hand-written forward and backward passes for accuracy. This initiative is significant for the AI/ML community as it emphasizes a transparent, from-scratch engineering approach, allowing researchers and developers to understand every component of the training process, from weight updates to the tokenizer. Despite its educational nature, NanoEuler demonstrates the potential of training models without heavy frameworks, though the output is currently limited to fluent but shallow text generation. The project serves as a proof of concept for the pretrain-fine-tune pipeline and contributes valuable insights into model behavior and training dynamics, while also highlighting the challenges of scaling model complexity and data for more advanced applications.