A Recipe for Training Neural Networks (2019) (karpathy.github.io)

The author argues that training neural networks is a “leaky abstraction” that looks easy from high-level APIs but hides many failure modes, and that these failures usually occur silently (e.g., flipped labels during augmentation, off-by-one autoregressive inputs, clipping the loss instead of gradients). To avoid hours of painful debugging, they propose a disciplined, hypothesis-driven process: build from simple to complex, verify each assumption with experiments, and be paranoid with visualizations and reproducibility. Concretely, start by thoroughly inspecting and visualizing the dataset (outliers, duplicates, label noise, class imbalance). Then implement a minimal end-to-end training/evaluation skeleton and validate it with dumb baselines: fixed random seed, no augmentation, full-test evaluation (not smoothed batch metrics), verify initial loss (e.g., -log(1/n_classes) for softmax), initialize final-layer biases to match label priors, train an input-independent baseline, overfit a single tiny batch, and confirm training loss decreases as capacity increases. Continuously visualize inputs immediately before the model and prediction dynamics during training. These practical checks force early detection of logical bugs, improve reproducibility, and make scaling to complex architectures far more reliable for ML practitioners.