Llama.cpp: 6.5x faster prompt processing on Arm with a NEON kernel (github.com)

The recent update to llama.cpp introduces a new NEON kernel for Arm CPUs, significantly improving prompt processing speeds for low-bit quantized models. Previously, Arm performance lagged behind x86 due to the lack of optimized inner kernels for certain data formats, forcing Arm to unpack weights for each dot product. With the integration of the NEON kernel, performance gains of up to 6.5 times faster have been measured across various Arm server cores, particularly with the IQ3_M format, highlighting its potential to enhance AI applications reliant on efficient computation. This upgrade is particularly noteworthy for the AI/ML community as it brings Arm-based systems closer to x86 performance levels, which could encourage broader adoption of Arm architecture in machine learning tasks. The technical enhancements include a unique design that utilizes dot-product instructions for bias correction and interleaved prompt processing, thus optimizing memory usage and execution efficiency. While the gains are impressive, the update does not affect generation speed and raises questions about how further optimization can be achieved, especially as workloads scale beyond the tested limits. This development marks a critical step towards optimizing AI workloads on diverse hardware platforms.