LLM Evaluation from Scratch: Multiple Choice, Verifiers, Leaderboards, LLM Judge (magazine.sebastianraschka.com)

The author published a practical, code-first guide that maps the four dominant ways people evaluate LLMs—multiple‑choice benchmarks, verifiers, leaderboards, and LLM judges—and provides from-scratch PyTorch implementations to illustrate each. The article walks through a hands‑on MMLU multiple‑choice example using a lightweight Qwen3 0.6B model (≈1.5 GB RAM), showing prompt construction, tokenization, generation, and a simple routine that extracts the first A/B/C/D token as the predicted answer. The accompanying reasoning_from_scratch library and GitHub examples demonstrate both token-level generation loops and alternatives like log‑prob scoring, and the writeup ties this material into the author’s early‑access book Build a Reasoning Model (From Scratch), which focuses on verifier-based evaluation. This matters because evaluation choice materially affects how we compare models and measure progress: multiple‑choice tests give clear, reproducible accuracy scores but only probe selection from fixed options; log‑prob scoring refines that by ranking candidate answers. Verifiers and LLM judges target reasoning and open-ended quality but introduce subjectivity and calibration issues; leaderboards aggregate results but can be gamed by dataset overlap or format tricks. The article’s emphasis on transparent, reproducible code helps practitioners understand trade-offs, implement consistent metrics, and choose evaluation methods that match their goals (knowledge recall vs. reasoning or human preference).