SnapLLM: Switch between local LLM in under 1ms Multi-model&-modal serving engine (github.com)

SnapLLM has been announced as a groundbreaking LLM inference engine that allows for sub-millisecond switching between multiple loaded language models, a significant leap forward from traditional approaches that take seconds to minutes for model transitions. Built atop llama.cpp and stable-diffusion.cpp, SnapLLM uses a vPID architecture to keep various models "hot" in memory, enabling instant switches in under 1ms. This engine supports a wide array of models, including text, vision, and diffusion types, employs a hybrid GPU/CPU capability for optimal performance, and is compatible with OpenAI’s API. The implications for the AI/ML community are profound. SnapLLM's rapid model-switching capabilities open new avenues for real-time applications in diverse domains, such as healthcare and law, where different models can be loaded and activated on-the-fly. The architecture’s efficient management of key-value caches also allows for O(1) query complexity, effectively enhancing performance while minimizing latency. Overall, SnapLLM not only streamlines the testing and deployment of multi-modal AI applications but also elevates the efficiency of AI-driven solutions across industries.