Writing an LLM from scratch, part 23 – fine-tuning for classification (www.gilesthomas.com)

Sebastian Raschka’s chapter on fine-tuning shows a pragmatic way to turn a pretrained LLM into a binary classifier by swapping out its output head — a technique the author dubs “decapitation” (a standard linear probe). Concretely, the model’s final projection from the 768-d embedding space to the full vocab is replaced with a 768→2 linear layer and trained with cross-entropy on the logits for only the last token (the token that carries the whole-sequence representation via attention). The chapter also stresses practical dataset and training hygiene: balance spam/ham across splits to avoid trivial baselines, set train DataLoader drop_last=True to keep batch sizes uniform, beware that dropout in frozen layers still runs in train mode (so Raschka uses dropout=0), and note sensitivity to random seeds. Empirically, freezing nearly all parameters (only the last transformer layer, final layer-norm and new head train) yields fast results (≈15s on an RTX 3090) with ~97% train / ~96% val / ~95.7% test accuracy, but occasional failures on clear spam examples. Fully fine-tuning all layers (≈42s) improved results to ~99.7% / 98.7% / 97.7%. Takeaways for practitioners: linear-probe decapitation is a quick, parameter-efficient classifier approach, but it can be seed-sensitive and less reliable than full fine-tuning; alternatives like LoRA or selective unfreezing give useful trade-offs between compute, stability, and final accuracy.