New in Llama.cpp: Model Management (huggingface.co)

The latest update to the llama.cpp server introduces a router mode that allows users to dynamically load, unload, and switch between multiple models without requiring a server restart. This lightweight, OpenAI-compatible HTTP server is designed for running large language models (LLMs) locally, and the new feature addresses popular demand for Ollama-style model management. With a multi-process architecture, each model operates independently, ensuring that a crash in one model does not affect others. This enhancement is significant for the AI/ML community as it streamlines the process of model switching, enabling developers to easily A/B test different model versions or conduct multi-tenant deployments. Key functionalities include auto-discovery of models from a specified cache or directory, on-demand loading, and least-recently-used (LRU) eviction of models to optimize memory use. Users can manage models via HTTP requests, allowing them to load or unload models seamlessly. This flexibility not only improves workflow efficiency but also supports more robust development practices without the downtime traditionally associated with restarting servers.