Writing an LLM from scratch, part 22 – training our LLM (www.gilesthomas.com)

In the final installment of a hands‑on series following Sebastian Raschka’s Build a Large Language Model (from Scratch), the author completes chapter 5 by actually training the tiny LLM they’ve built and then loading OpenAI’s original 124M GPT‑2 weights. Training on a toy dataset (20k characters of Edith Wharton’s The Verdict) produced a rough but coherent next‑token sample in about eleven seconds on an RTX 3090, while swapping in the 124M GPT‑2 weights generated a far more fluent continuation. Cross‑entropy loss and perplexity were the toughest concepts to internalize; after that the chapter largely becomes “plug and run” code that reveals learning curves and model behavior in practice. The post highlights several practical, technical takeaways valuable to ML practitioners: reproducibility is tricky because many helper functions use randomness and strict torch.manual_seed ordering is brittle; AdamW is the pragmatic optimizer choice (zero_grad → loss.backward() → optimizer.step()); and sampling tricks—softmax sampling, temperature scaling (divide logits before softmax) and top‑k filtering (set logits below the k‑th largest to −∞)—control creativity versus parroting. The author also underscores the enormous speed gap between CPU and GPU training (MacBook Air vs RTX 3090) and notes nuances like GPT‑2 embedding sizes (124M uses 768). Next steps include testing larger training runs, measuring tokens/sec, and experimenting with fine‑tuning and deployment cost tradeoffs.