AutoUVM: Automated Prefetching Framework for LLMs Under UVM Oversubscription (arxiv.org)

Researchers have introduced AutoUVM, an innovative framework designed to optimize the prefetching process for large language models (LLMs) operating under NVIDIA's Unified Virtual Memory (UVM) system, which is often subject to memory oversubscription. As LLMs frequently exceed the memory capacity of standard GPUs, the traditional UVM approach can lead to significant performance slowdowns due to page-fault-induced memory migrations. AutoUVM aims to overcome these limitations by facilitating policy-driven prefetching at the tensor level, directly addressing the specific memory behaviors of deep learning frameworks without requiring any changes to existing model code. The significance of AutoUVM lies in its ability to enhance LLM performance dynamically, achieving an impressive average speedup of 3.1x compared to the standard UVM and surpassing existing prefetching methods by noteworthy margins. It achieves this efficiency by intelligently responding to runtime memory demands and prioritizing crucial data transfers, thus alleviating the pressure on CPU-GPU interconnects. With improvements noted across a range of ten LLMs, AutoUVM not only sets a new benchmark for memory management in deep learning environments but also paves the way for broader applicability of UVM by making it more user-friendly for AI practitioners.