Designing a TPU and optimizing transformer inference on a $100 FPGA (im-afan.github.io)

A summer project has successfully enabled the operation of a small transformer model on a Cmod A7 FPGA board, marking a significant exploration into machine learning (ML) performance optimization and hardware-software co-design. By leveraging a matrix multiply unit (MXU) and vector processing unit (VPU) within the FPGA, the project aims to analyze and optimize transformer inference while circumventing strict architectural constraints. This initiative is particularly noteworthy as it allows for the testing of various model architectures and the fine-tuning of operation efficiencies, a critical consideration given the prevalent challenges in compute and communication bottlenecks within ML workflows. The project emphasizes the significance of matrix multiplications and elementwise operations, with particular attention to quantizing weights and activations to int4 to maximize resource usage. Using a tiling strategy in matmul operations, the design effectively manages memory between the scratchpad (on-chip cache) and external SRAM, facilitating higher throughput operations. The architecture incorporates a flexible programming scheme enabled by the PicoRV32 softcore processor, which allows for easy testing and optimization of different algorithms. The work bridges theoretical concepts from model scaling and TPU architecture, demonstrating practical applications that contribute valuable insights into enhancing transformer inference performance, thus benefiting the broader AI/ML community.