Photoroom cut inference cold start to seconds with Nvidia Dynamo (www.photoroom.com)

Photoroom has significantly reduced GPU inference cold starts from approximately 220 seconds to just 35-45 seconds by utilizing NVIDIA's Dynamo Snapshot. This innovative approach leverages NVIDIA's cuda-checkpoint and CRIU (Checkpoint/Restore In Userspace) to snapshot a fully-loaded process and restore it on any GPU within a Kubernetes ecosystem. This advancement is crucial for the AI/ML community as it addresses the well-known cold start problem associated with scaling to zero, which can hinder performance for applications that rarely receive requests. At Photoroom, various computer vision and generative AI models each have their own latency requirements. With the integration of GPU memory checkpointing and NVIDIA Dynamo, the process of restoring a model from a snapshot bypasses the time-consuming steps typically required, such as library imports, model downloads, and warm-ups. The Dynamo Snapshot framework facilitates efficient management of LLMs (Large Language Models) by enabling quick process restoration, which is especially beneficial in a dynamic and autoscaled environment. This capability is poised to enhance the performance of AI applications, making them more responsive and cost-effective in production settings.