70x faster cold(ish) starts for SGLang (fergusfinn.com)

SGLang has announced a substantial improvement in cold start times for its model serving, achieving a 70x reduction in startup duration for the 122 billion parameter MoE model Qwen3.5-122B-A10B-FP8. Initial cold starts took nearly twelve minutes, largely due to prolonged weight loading and overhead from Python imports, autotuning, and kernel compilation. Through caching and optimization of these processes, including adopting checkpoint/restore mechanisms with CRIU, SGLang's new approach has cut initial start times to approximately 9.6 seconds, approaching theoretical limits for efficient memory and data transfer. This development is particularly significant for the AI/ML community as it addresses one of the critical bottlenecks in deploying large-scale machine learning models—startup latency. The technical implications include the use of GPU Direct Storage and innovative techniques for memory management, allowing for faster loading of model weights and significant reductions in overhead time during model instantiation. By storing only necessary data in checkpoints and leveraging RAM for quicker access, SGLang enhances workflow efficiency and responsiveness in environments like Kubernetes, setting a new standard for rapid model deployment in AI applications.