Show HN: TiGrIS, a tiling compiler that fits ML models onto embedded devices (github.com)

TiGrIS, a new tiling compiler, has been introduced as a solution for fitting machine learning (ML) models onto memory-constrained embedded devices. Unlike traditional methods that typically involve shrinking models through quantization or pruning, TiGrIS employs an innovative approach by partitioning the compute graph of an ONNX model and tiling spatial operations. This allows it to create a flat binary plan that executes without dynamic memory allocation, effectively minimizing memory usage. For instance, TiGrIS can compress a model with a naïve peak memory requirement of 4.59 MiB down to just 256 KiB of SRAM, demonstrating its efficiency. This development is significant for the AI/ML community as it opens up new possibilities for deploying complex ML models on low-resource hardware, expanding the applicability of AI in edge devices where memory is at a premium. The TiGrIS compilation process involves a few simple steps: analyzing model feasibility, compiling it to a binary plan with weights stored in flash, and generating a backend-specific C harness for integration. This versatility, combined with support for various kernel backends, allows developers to easily adapt their models for different embedded platforms, promoting greater accessibility and innovation in AI applications within constrained environments.