Reverse-Engineering the RK3588 NPU: Hacking Limits to Run Vision Transformers (amohan.dev)

A recent project has successfully reverse-engineered the RK3588 NPU to make it compatible with Vision Transformers, showcasing an innovative approach to overcoming the hardware's limitations. Initially, attempts to run the SigLIP Vision Transformer on the chipset led to performance issues and errors due to the standard Computer Vision SDK's optimization for older CNNs. By employing a "First Principles" approach and digging into undocumented errors, the engineer discovered that a hardware-enforced memory limit was causing the failure. This prompted the development of a "Nano-Tiling" algorithm that efficiently partitions data into manageable chunks, circumventing the memory constraints. The implications of this breakthrough extend beyond just running a specific model, as it illustrates how to leverage constrained hardware for modern AI applications. By introducing a "Poison Pill" to prevent compiler optimization from crashing the hardware, and implementing a custom runtime scheduler to distribute tasks across multiple NPU cores, the project achieved impressive results, reducing inference time from 30 seconds to under 1.8 seconds while maintaining high accuracy. This work challenges assumptions about "supported hardware" in AI and could inspire engineers to further explore and optimize lesser-known edge accelerators for advanced machine learning tasks.