LLM for the ESP32-S3 (github.com)

A new breakthrough in AI was announced with the successful splitting of a Llama-architecture language model (LLM) across two ESP32-S3 microcontrollers. This innovative approach allows for the handling of larger models, specifically a 15 million parameter model operating at approximately 1.4 tokens per second, with plans to support even larger models in the future. This marks a significant advancement as it is the first multi-chip pipelined LLM inference implemented on ESP32-class hardware, pushing the boundaries of what can be achieved with affordable microcontroller technology. The technical ingenuity lies in the communication protocol and deployment strategy. By linking the chips through a CRC-framed UART interface, the researchers effectively distribute weight storage across the combined flash memory, effectively bypassing individual chip limitations. Weights are streamed using integer quantization techniques, resulting in minimal RAM usage and employing optimized computation methods. The system supports rigorous testing to ensure reliability, matching outputs to a NumPy reference with exceptional precision. This development not only opens the door to more complex models on low-power devices but also exemplifies a practical step forward in making powerful language models accessible for various applications in the embedded systems domain.