Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20 (github.com)

Andrej Karpathy demonstrated that you can reproduce GPT-2 (124M) from scratch using llm.c — a compact ~4,000-line C/CUDA training implementation — very cheaply and quickly: about $20 on a rented 8x A100 80GB node (~90 minutes) or on a single GPU with proportionally longer runtimes (hours). He trained a 12-layer, 12-head, 768-dim Transformer on a 10B-token FineWeb corpus (seq length 1024) and reports outperforming the original GPT-2 124M checkpoint on the FineWeb validation split and reaching 29.9 HellaSwag accuracy (vs GPT-2’s 29.4; GPT-3 Small cited at 33.7 but was trained on far more data). The run demonstrates llm.c’s practical efficiency (model FLOP utilization ≈50–60% on A100s) and token throughput (~178K tok/s on an A100 40GB), with training logs, tokenizers and reproducible scripts provided. Key technical takeaways: llm.c implements Transformer forward/backward in CUDA (with optional cuDNN/FlashAttention), supports MPI multi-GPU, and exposes hyperparameters that follow GPT-3-style optimization. The repo includes data preprocessing (FineWeb), build and training commands, and sampling hacks. Karpathy also scaled a 350M run (30B tokens, ~$200 on 8x A100) and estimates ~1 week / ~$2.5K to reach 1.55B params on the same single-node box. This work highlights how efficient, open training stacks lower the barrier to reproducing and iterating on classic LLMs, enabling affordable experimentation and faster research cycles.