Running a 28.9M parameter LLM on an $8 microcontroller (github.com)

A groundbreaking 28.9 million parameter language model has been successfully run on the ESP32-S3 microcontroller, which costs just $8. This achievement marks a significant leap in the AI/ML community, as previous models of this size were limited to only 260,000 parameters on similar hardware. By utilizing Google's innovative Per-Layer Embeddings strategy, the majority of the model's parameters are stored in slower flash memory rather than the limited fast SRAM, allowing for efficient processing without overloading the chip's capabilities. The model executes at approximately 9 tokens per second, generating coherent short stories using a dataset called TinyStories. This development underscores the possibility of deploying advanced language models on inexpensive, low-resource devices, expanding access to AI functionalities and enabling innovative applications in embedded systems. While the model's reasoning capacity is limited and it cannot perform complex tasks like answering questions or coding, its primary significance lies in demonstrating how to fit larger AI models into smaller footprints. The firmware and training materials related to this project are publicly available, further encouraging experimentation and collaboration within the AI community.