Teaching an M4 CPU to Tell Stories with XOR, Popcount and SDOT (dnhkng.substack.com)

A new approach has emerged in language model design, leveraging ARM M4 CPU architecture to enhance performance through specialized instructions such as XOR, popcount, and signed dot products (SDOT). This innovative technique combines traditional linear algebra with low-level hardware optimizations, resulting in a 5-layer, 512-wide TinyStories model. The model achieves an impressive throughput of 20,882 tokens per second while maintaining a byte-level efficiency of 0.8786 bits per byte, which is notable for its ability to optimize resource usage on hardware not originally designed for AI applications. This development is significant for the AI/ML community as it illustrates a novel path for efficient model deployment on existing hardware, emphasizing the interplay between software and silicon architecture. By harnessing the unique capabilities of ARM's instruction set, the project demonstrates that efficient training and inference in language models can benefit from tailored CPU instructions, enabling better computational efficiency and potentially reducing energy consumption. The model's structure allows for flexible representation where both numeric and Boolean operations can be seamlessly integrated, paving the way for future explorations into hardware-software co-design in machine learning. The complete implementation and code are available for further study at the provided GitHub repository.