Making a vintage LLM from scratch; Take #2 (crlf.link)

A developer has made significant advancements in creating a vintage language model (LLM) trained exclusively on texts written before 1900, following an earlier attempt that produced a model unable to engage in conversation. Over the past three months, the creator has meticulously curated new datasets, constructed three new vintage models, and implemented an evaluation pipeline, leading to a current focus on fine-tuning. Key accomplishments include the creation of enhanced datasets, like the comprehensive Sprocket-n-Say, incorporating 15 million rows of high-quality vintage texts, and improvements in model performance by addressing the challenges of training data quality. This project is noteworthy for the AI/ML community as it highlights the importance of dataset quality in training effective models. The developer has navigated challenges such as OCR noise and the need for synthetic data, employing techniques that involve modern LLMs to clean and generate training content. Additionally, the undertaking raises ethical concerns regarding the treatment of historical texts in AI development, invoking a discussion about the preservation of literature versus the needs of large-scale AI training. The complete code and models are open-source and shared on platforms like GitHub and Hugging Face, fostering community engagement and collaboration.