Llama.cpp flags auto-tuning tool (github.com)

The recent announcement of ggrun, a new auto-tuning tool for launching large language models (LLMs), marks a significant advancement in the AI/ML landscape. Formerly known as llm-server, ggrun simplifies the process of configuring complex model parameters by automatically measuring GPU configurations, RAM, and PCIe topology. It selects the most efficient backend—either llama.cpp or the faster ik_llama.cpp—while ensuring precise multi-GPU and mixture of experts (MoE) placements tailored to the specific hardware. Notably, this allows for effective loading of large models across mismatched GPUs, dramatically enhancing performance. The tool's auto-tuning capabilities lead to a considerable performance boost, with ggrun achieving 49–74% faster inference speeds compared to competitors like Ollama on various models. This improvement stems not only from backend selection but also from its advanced management of KV-cache types and batch sizes based on real-time measurements. Additional features include a user-friendly terminal UI, hardware-aware model downloading from Hugging Face, and options for speculative decoding and crash recovery. By automating what traditionally required extensive manual configuration, ggrun empowers researchers and developers to maximize their hardware's potential, facilitating more efficient experimentation and deployment of AI models.