Reverse Engineering Google's TPUs (gherkinlabs.com)

Gherkin Labs has successfully reverse-engineered Google's Tensor Processing Units (TPUs) and developed a simulator that replicates the real TPU software stack. This simulator allows users to submit JAX programs and receive detailed cycle-level traces of how the TPU would execute them. This breakthrough is significant for the AI/ML community because TPUs, while powerful, have been challenging to leverage effectively due to sparse public documentation. By illuminating the TPU's communication protocols and inner workings, Gherkin Labs aims to enhance the performance and usability of machine learning tasks on these chips. The simulator, named libsymtpu.so, mimics TPU functionalities on a CPU, enabling large topologies to be run with minimal RAM. It provides cycle-accurate insights by replaying compiler schedules against the TPU’s hardware architecture, which consists of tensor cores and sparse cores designed for different computational tasks. This low-level engineering approach not only reveals how to maximize TPU performance but also sets the stage for future innovations in kernel engineering, particularly with LLM-driven optimizations. As demand for efficient machine learning accelerators rises, this work could pave the way for broader accessibility and understanding of TPUs beyond Google's ecosystem.