Retro Language Models: Rebuilding Karpathy's RNN in PyTorch (www.gilesthomas.com)

This post rebuilds Andrej Karpathy’s classic character/byte-level RNN in PyTorch (using torch.nn.LSTM) and publishes a hands-on implementation and walkthrough to make the old-school approach easy to inspect and run. The author stays close to Karpathy’s original Torch code but adapts it to PyTorch idioms, and promises follow-up posts that implement custom RNNs (rather than using the built-in LSTM). The repo and code walk-through are organized around a NextByteDataset/NextByteTokenizer setup that mirrors the training loop Karpathy described. Technically the write-up highlights the biggest practical differences between RNN training and modern Transformer LLM training: you must use truncated backpropagation through time (TBPTT) to avoid vanishing/exploding gradients, keep continuous sequences for each batch position (no random shuffling of chunks), and validate via vertical splits of those sequential streams. The NextByteDataset enforces seq_length (Karpathy often used 100), computes num_sequences = (len(full_data)-1)//seq_length, trims the data, tokenizes unique bytes into compact IDs (instead of naive 256-d one-hots), and returns x_ids/y_ids tensors (plus raw bytes for debugging). The post is useful for practitioners wanting a reproducible, minimal RNN baseline, a clearer view of TBPTT/batching implications, and a concrete comparison point to Transformer-style LLM training.