Measured LLM inference speeds on Apple Silicon, with raw data (CC BY 4.0) (macyou.co)

Recent benchmarks reveal the impressive inferential speeds of large language models (LLMs) on Apple Silicon, specifically the M4 Mac mini, emphasizing the hardware's capabilities in AI applications. The testing employed the Ollama API, yielding specific generation and prompt processing rates across several models. For instance, the Llama 3.2 3B model achieved a generation speed of 46.7 tokens per second, while larger models like the 14B Qwen 2.5 demonstrated lower but still noteworthy rates, almost entirely influenced by the silicon's memory bandwidth. This data signifies a pivotal moment for developers and researchers in the AI/ML community as it provides concrete performance metrics, greatly informing decisions on hardware selection for LLM deployments. The benchmarks indicate that speed scales with the chip's bandwidth, suggesting that forthcoming Apple Silicon variants, such as the M4 Pro and M3 Ultra, are likely to enhance performance further. This ongoing assessment broadens the understanding of executing LLMs on ARM architecture and could shape the future of scalable AI solutions, particularly in environments where efficient processing and rapid inference are crucial.