I trained a 113M-parameter earthquake LLM from absolute scratch (github.com)

A researcher has successfully developed a 113-million-parameter language model (LLM) tailored for earthquake science, named nanoGPT-Seis, from scratch using a meticulous, step-by-step approach. The entire lifecycle of the model, including data crawling, cleaning, tokenization, training, and inference, was documented to enhance transparency and educational value. This project utilized two NVIDIA A30 GPUs and leveraged various sources, combining specialized earthquake texts with general language content, resulting in a corpus of over 500,000 documents. The model boasts a context length of 4096 tokens and a high fluency score, demonstrating the effectiveness of integrating general text for improved language generation. This initiative is significant for the AI/ML community as it serves not only as a functional model for earthquake-related inquiries but also as a comprehensive teaching resource that demystifies the LLM training process. Observations from the training revealed that longer context windows improved model performance, while the incorporation of general knowledge sources enhanced fluency, addressing common issues of repetitiveness and incoherence in domain-specific models. The project's repository is available on Hugging Face, providing valuable insights and tools for practitioners interested in similar LLM development and training practices.