Swapping LLMs in Under Two Seconds: A Sleep Mode for Llama.cpp (zaage.it)

A new development in the AI/ML community introduces llama-sleep, a wrapper that significantly reduces the model swapping time for large language models (LLMs) from over 10 seconds to as little as 1.7 seconds. By enabling a "sleep mode" that parks a model's GPU memory in host RAM instead of fully unloading it, llama-sleep improves efficiency and keeps cached prompts intact during swaps. This advancement allows tasks requiring different models to be executed more fluidly, drastically enhancing responsiveness in applications needing quick model transitions. The implementation leverages NVIDIA's cuda-checkpoint functionality and employs multi-size transparent huge pages, which optimize memory management during the swapping process. With minimal additional software complexity, the solution maintains model integrity and smoothens operation by holding requests while the target model wakes up. This innovation marks a significant leap in managing language models, particularly for users reliant on two or more models per GPU for tasks like voice assistance and automation. The ability to exchange models swiftly without compromising on the performance of cached data could set a new standard for resource management in AI applications.