Tiny TPU (www.tinytpu.com)

Tiny-TPU is an open, toy implementation of a Google‑style TPU built by a team of novices who deliberately re‑invented the accelerator from first principles as a learning project. They implemented both inference and training for a small MLP solving the XOR “hello world” problem, using Verilog to model clocked hardware. Rather than a full TPUv1’s 256×256 array, they scaled the design to a 2×2 systolic array of Processing Elements (PEs) that perform multiply‑accumulate (MAC) each clock cycle, and added supporting modules: input/weight FIFOs with staggered scheduling and transposition, bias broadcast units, per‑column Leaky ReLU activations, and pipelining. The post explains matrix input formats (e.g., 4×2 batch for XOR), weight loading, and the dataflow that moves activations right and partial sums down through the array. This project matters because it provides a concrete, pedagogical walkthrough of how matrix multiplications—which dominate transformer and CNN compute—map to silicon. By showing PE behaviour, scheduling tricks (rotation, staggering), FIFO/accumulator designs, and simple Verilog snippets, Tiny‑TPU gives engineers and researchers a practical reference for accelerator dataflow, hardware/software co‑design, and training on ASIC‑like substrates. It’s explicitly not a 1:1 TPU replica, but a faithful, scalable distillation that highlights the key tradeoffs (latency, pipelining, weight reuse) and lowers the barrier to understanding and prototyping ML accelerators.