AI is now capable of developing its own inference hardware (github.com)

A notable advancement in the AI hardware space has been achieved with the announcement of openTPU, an open-source AI accelerator developed by AI. This project leverages the principles of auto-architecture and explores the potential of AI agents in designing custom hardware for their own inference tasks. OpenTPU encompasses a comprehensive monorepo that includes the hardware architecture, instruction set, bit-exact simulator, kernel language, compiler, and software to operate a PCIe card, providing an invaluable resource for understanding AI accelerator functionalities from the ground up. The significance of openTPU lies in its demonstration of AI’s capability to innovate in hardware design, potentially democratizing and streamlining the development process of AI hardware. The accelerator can successfully run ten modern models, achieving high throughput on the Inspur YPCB-00338 card. For example, the LFM2.5 model operates at 59.0 tokens per second using int8 weights, and performance improves by up to 45% with 4-bit quantization. These advancements highlight both the efficiency gains in model inference and the critical role of custom hardware in optimizing AI workloads, paving the way for more sophisticated AI applications and further research in AI-centric hardware design.