From Matmul to Meaning (www.evis.dev)

Over the past few months the author built two LLMs from scratch (GPT-rs and llama2.rs) and published a clear, intuition-first explainer called “From Matmul to Meaning” that traces how the humble matrix multiplication underpins everything transformers do. The piece starts with 2D vectors and basis vectors, shows how matrices are collections of transformed basis directions, then walks through matrix–vector and matrix–matrix multiplication as repeated dot products. It emphasizes that transformers perform these matmuls trillions of times, and that each multiplication is just scaling columns (directions) by input coordinates and summing to project tokens into new semantic spaces (x · W = h). This matters because it connects low-level linear algebra to high-level semantic behavior: dot products measure alignment (similarity), learned weight matrices implement reusable linear transformations (e.g., a “change gender” mapping), and simple vector arithmetic like king − man + woman = queen emerges naturally when dimensions encode semantic axes. The post argues why matrix multiplication — not elementwise ops — is essential: it preserves relationships across vectors and applies the same transformation broadly, enabling consistent, scalable representation learning across thousands of dimensions and billions of parameters. For practitioners, this is a compact, practical bridge from matmul mechanics to why LLMs capture meaning.