The Little Book of llm.c – friendly explaining llm.c in plain English (github.com)

The Little Book of llm.c is a plain‑English companion to Andrej Karpathy’s compact llm.c reference implementation of GPT-2. It walks readers step‑by‑step through the repo — from build flags and file layout to running CPU and CUDA training (train_gpt2.c, train_gpt2_fp32.cu, train_gpt2.cu) — explaining core concepts like tokenization, binary dataset format, data loaders, batching, checkpointing, seeding and reproducibility. Designed for beginners and practitioners, the guide demystifies the full training stack (forward pass, attention QKV, MLP, LayerNorm, residual streams, tied embeddings and loss head) and includes practical tooling: OpenMP pragmas, profiling, example logs, and tests that compare CPU/CUDA runs to PyTorch. Technically rich yet accessible, the book covers optimizer and training mechanics (AdamW, LR schedulers, gradient accumulation), GPU details (cuBLAS GEMMs, FlashAttention kernels, mixed precision FP16/BF16, loss scaling, activation checkpointing), and multi‑GPU/multi‑node patterns (MPI, NCCL, SLURM). It also documents performance measurement (FLOPs), common CUDA and data pitfalls, and how to extend the code (custom kernels, new optimizers, dataset pipelines), plus a porting playbook (Go, Rust, Metal). With reproducible recipes for scaling GPT‑2 up to 1.6B (example: 8×H100) and open invites for contributions, the book is a practical learning bridge from high‑level ML frameworks down to transparent, minimal C/CUDA LLM engineering.