Let's Build the GPT Tokenizer: A Complete Guide to Tokenization in LLMs (www.fast.ai)

Andrej Karpathy’s 2h13m tokenizer lecture has been transcribed into a concise, book‑chapter style tutorial (code and key images included) using the Solveit workflow — a human‑led effort accelerated with AI help that the authors say was ≈10x faster than doing it manually. The chapter reproduces Karpathy’s lesson, links to video timestamps, and promises a full write‑up of the conversion process next week. Importantly, this is a carefully checked human product (not a pure auto‑generation), designed to teach tokenization from first principles and include runnable examples. The tutorial emphasizes why tokenization is foundational to LLM behavior: many quirks (spelling, string processing, arithmetic, non‑English performance, odd halting or code errors) often trace back to how text is split into tokens. It contrasts naive character‑level tokenization (small vocab, simple integer mapping, one token per char) with real‑world schemes like byte‑pair encoding (BPE) used in GPT‑2 (vocab ≈50,257, context 1,024) and modern models, and explains practical issues — UTF‑8 vs. raw Unicode code points, vocabulary size tradeoffs, and embeddings (tokens → row lookups → vectors fed into transformers). The chapter demos token fragmentation (via the tiktoken web app) where numbers/words split unpredictably, and shows why encodings and BPE matter for model robustness, multilingual support, and downstream behavior.