Getting 50 GB/S Back Out of the Neural Engine (eiln.github.io)

A recent analysis of the Apple M3 Neural Engine (ANE) has revealed a significant performance issue affecting the throughput of DRAM weight streaming. Due to an RTL performance erratum, the throughput is throttled to 17–19 GB/s when weights are integer multiples of 1 MiB, down from the expected 45–60 GB/s. This issue impacts seven of ANEMLL’s 15 models, illustrating a broader efficiency challenge for neural network processing on Apple’s hardware. Innovative profiling has demonstrated that modifying the dimensional configuration of token decoding can improve throughput markedly, as seen with Llama 3.2, which experienced a jump from 10.0 to 24.3 tokens/s by sidestepping the problematic path in the kernel DMA engine. This discovery is significant for the AI/ML community, especially developers and researchers utilizing Apple’s neural architecture, as it could influence software optimization strategies. The analysis included sweeping through various tensor dimensions and core configurations, revealing a consistent drop in bandwidth at powers of two, leading to the hypothesis that core contention at specific bandwidths might be causing the issue. This throttling was evident regardless of the number of active cores, suggesting that careful attention to weight tensor sizes and memory access patterns could mitigate the performance drop. Overall, these findings underscore the importance of memory architecture in optimizing AI models and could prompt further exploration into DRAM access patterns within neural network frameworks.