Retrospectively Reverse-Engineering Apple's Neural Engine (eiln.github.io)

A recent endeavor to reverse-engineer Apple's Neural Engine (ANE) has revealed significant insights into its architecture and implications for AI workloads. Initially deemed ineffective for diverse applications, the ANE's architecture has been found to cater primarily to dense image-processing tasks associated with earlier Convolutional Neural Networks (CNNs). The shift in Apple’s newer products, notably the M5 integrating ANE functions into GPUs for large language model (LLM) processing, hints at the potential obsolescence of standalone neural processing units. This reverse-engineering project aims not only to unlock the ANE's full internal structure but also to illuminate Apple’s design choices and assumptions about machine learning workloads since the A11 Bionic chip's inception in 2017. The ANE features 16 compute cores, each with 128 FP16 or 256 INT8 parallel multiply-accumulate (MAC) lanes, enabling efficient processing for specific tasks. However, the evaluation suggests that while the basic operations like dot products and tensor reductions remain pivotal, the real innovation lies in the dataflow architecture that supports modern transformer models, which break previous assumptions of predictable reuse. This investigation highlights the processor's potential in adapting to current AI demands, especially as future architectures may need to reconcile legacy structures with evolving computational paradigms in machine learning.