Visual Information Theory – colah's blog (colah.github.io)

Colah’s “Visual Information Theory” reframes core information-theoretic ideas with intuitive diagrams: probability grids that show independence as straight vertical/horizontal lines, conditional distributions as reshaped cells, and factorization p(x,y)=p(x)p(y|x) (and its Bayes-flipped form). He uses concrete mini-examples — sunny 75%/rain 25%, coat 38%, giving p(rain,coat) ≈ 19% — and a visual 3D split to demystify Simpson’s paradox (apparent treatment reversal from confounding by stone size). These visuals make why conditioning, marginalization and causal structure matter immediately clear for ML practitioners dealing with biased datasets or model interpretation. The post then moves to coding: mapping symbols to bit sequences and visualizing codeword probability p(x) against length L(x) so area = expected code length. A uniform 2-bit code can be improved by assigning shorter codewords to frequent symbols (Bob’s “dog” example), reducing average length to 1.75 bits; that reduction is bounded below by the distribution’s entropy — a fundamental limit. Colah highlights key trade-offs (shortening common codewords forces rarer ones to grow), the exponential growth of code-space with length (2^L), and the decoding constraint that motivates prefix-free codes and the Kraft-type counting arguments. The result is a visually grounded, practical entry point to entropy, coding theory and common statistical pitfalls relevant to AI/ML.