ESP32S3 cluster running 1.58-bit (BitNet) Language model (github.com)

A groundbreaking project has successfully distributed the 1.58-bit BitNet language model across a cluster of seven ESP32S3 microcontrollers, showcasing an innovative approach to running large language models on resource-constrained devices. The architecture features a master node responsible for tokenization and embedding, while the compute nodes handle attention layers and multi-layer perceptrons, all communicating through a high-speed SPI daisy-chain. This setup allows for efficient processing of a sliced 0.5 billion parameter model, demonstrating the potential of low-power devices in AI applications. This development is significant for the AI/ML community, as it highlights how compact and energy-efficient hardware can enable practical implementations of language models traditionally limited to more powerful computing environments. The technical details, including specialized implementations of 1.58-bit attention and traditional FP16 and INT4 data formats, enhance the model's performance while minimizing resource usage. By employing quantization-aware training and optimized assembly code for operations, this project paves the way for wider adoption of similar architectures in edge computing and IoT applications, where power and computational efficiency are paramount.