Bits-per-Byte (BPB): a tokenizer-agnostic way to measure LLMs (dipkumar.dev)

Karpathy’s nanochat repo uses Bits-Per-Byte (BPB) as an alternative to per-token cross-entropy or perplexity. BPB is simply the model’s cross-entropy (in nats) converted to bits by dividing by ln(2), then normalized by the total number of UTF-8 bytes those target tokens represent: BPB = sum_nats / (ln(2) * total_bytes). Practically, the implementation sums per-token losses (loss_reduction='none'), looks up each token’s byte-length via a token_bytes vector (0 for specials), excludes ignored/special tokens, and computes a distributed all-reduce over losses and byte counts before the final division. Why it matters: token-level metrics (avg CE, perplexity) change when you swap tokenizers—BPE vs. Unigram, different vocab sizes, special tokens, merge rules—because the same text can split into more or fewer tokens. BPB is tokenizer-agnostic: it measures how many bits the model needs to encode the raw UTF-8 text, letting you compare models fairly across differing tokenizers and vocabularies. Key implications: more stable benchmarking across architectures and tokenization schemes, easier comparison of models that use cl100k, TikToken, SentencePiece, etc.; caveats include reliance on accurate token->byte mapping and that models still operate in token space, so BPB summarizes token predictions in byte terms rather than changing the modeling objective.