Benchmarking Local LLM Servers: Llama.cpp, Llamafile, LM Studio, and Ollama (blog.mozilla.ai)

Four local LLM servers—llama.cpp, LM Studio, Ollama, and llamafile—were benchmarked to evaluate their performance differences using a custom tool, exp-llama-benchy. The tests were conducted on three platforms: a Mac Studio M4 with Metal GPU acceleration, a Linux server with an NVIDIA L40S using CUDA, and a Steam Deck via Vulkan, across varying model sizes and token configurations. The results indicated that configuration choices greatly influenced speed, with specific optimizations such as enabling CUDA graphs improving performance by over 16% on the L40S. Notably, while the servers shared a common core functionality, prompting varied performance based on environmental factors, where draft settings for speculative decoding played a crucial role. This benchmarking is significant for the AI/ML community as it emphasizes that the selection of server software and its configurations can yield substantial differences in performance, potentially impacting deployment strategies for applications in document processing, interactive chats, or code generation. The study revealed that most decoding performance impairments stemmed from fixed overhead costs rather than GPU discrepancies, highlighting the importance of optimally configuring server settings to match specific workloads. Overall, these findings provide valuable insights for developers and researchers looking to maximize the efficiency of local model deployments.