Training a 3.8B LLM to 0.384 CORE for $998 – Hugo Vergnes (hugovergnes.github.io)

Hugo Vergnes has successfully trained a 3.8 billion-parameter language model, little-lm, scoring an impressive 0.384 on the CORE benchmark in just 43 hours for a total cost of $998. This project highlights a significant milestone in the AI/ML community by demonstrating that individuals with modest budgets can still achieve meaningful results without the need for extensive resources typically found in research labs. The model was built using a config-driven framework, allowing for simple adjustments through a YAML file, which promotes efficiency and repeatability in training experiments. Vergnes employed several innovative strategies that contributed to the model's performance, including a trapezoidal learning rate schedule and using FP8 precision to enhance throughput. By optimizing various components and experimenting with configurations, he effectively used hardware resources—evidenced by the model achieving about 92% streaming multiprocessor (SM) activity during training. This work illustrates the growing accessibility of large-scale model training and suggests that as computational capabilities expand, breakthroughs can emerge from smaller setups, empowering a broader range of individuals to contribute to the field of AI.