Ollama replacement 2-4x faster for no extra compute cost (github.com)

A new server called llmash, compatible with Ollama and built on llama.cpp, has been announced, boasting performance improvements of 2-4x without additional compute costs. This server allows users to serve GGUF files and maintain existing commands, APIs, and model stores associated with Ollama, while optimizing llama.cpp settings per model at launch. It simplifies the setup process by automatically fetching the required llama.cpp build for the user's GPU and streamlining operation through a command line. The significance of llmash lies in its enhanced performance metrics, which demonstrate substantial improvements in token processing speeds for models such as gemma-4 and Qwen. For instance, gemma-4 achieved an increase from 244 to 342 tokens per second, while Qwen3.6 surged to 382.5 tokens per second in various tasks, leveraging advanced options like speculative decoding and prompt-prefix reuse. These enhancements provide developers and researchers with a powerful tool for optimizing inference without incurring additional costs, potentially accelerating AI and ML projects considerably.