LLaMA Now Goes Faster on CPUs (justine.lol)

A recent announcement from developer Justine reveals significant performance enhancements for local language models (LLMs) using the llamafile framework, which now features 84 newly optimized matrix multiplication kernels. These improvements boost prompt evaluation speeds on CPUs by 30% to an impressive 500%, particularly for users of ARMv8.2+ and modern Intel architectures. The kernels can achieve up to twice the speed of Intel's Math Kernel Library (MKL) for matrix operations fitting in L2 cache, although the speedup peaks with prompts under 1,000 tokens. This advancement is vital for the AI/ML community as it democratizes access to enhanced LLM performance on relatively low-cost hardware, such as the Raspberry Pi 5. The optimizations capitalize on new ARMv8.2 instructions, making model execution faster and less resource-intensive without the need for high-end GPUs. Similar enhancements on Intel’s Alderlake processors reveal that LLMs can perform tasks like spam filtering significantly faster than on previous architectures, underscoring the ongoing relevance of local models in practical applications. This initiative not only bolsters the performance of llamafile but also aims to improve the user experience for those in the AI field, enhancing the accessibility of advanced LLM capabilities across diverse computing environments.