Time to build a GPU OS? Here is the first step (www.notion.so)

Researchers from UC Berkeley, Rice and UCLA released kvcached, a C++ PyTorch plugin that virtualizes LLM KV caches to enable elastic, multi-tenant GPU memory sharing. kvcached leverages recent CUDA Virtual Memory Management (VMM) APIs to allocate virtual address ranges for KV caches without immediately backing them with physical pages, then maps/unmaps GPU pages on demand. Unlike PagedAttention—which still requires static physical reservations per process—kvcached provides a “Virtual Tensor” that is API-compatible with torch.Tensor and directly usable by CUDA kernels, letting serving engines (e.g., SGLang, vLLM) run multiple models on the same GPU and reclaim idle memory dynamically. The team also built runtime patching so many engines can adopt kvcached with little or no code changes. Technically, kvcached addresses fragmentation and compatibility challenges: a fragmentation-aware KV cache manager aligns allocations with 2MB GPU physical pages, prioritizing partially used pages to reduce waste; an asynchronous preallocation thread maintains a buffer of ready pages to cut latency; and a zero-page mechanism ensures safe CUDA-graph execution by mapping unmapped virtual ranges to a shared read-only zero page. These design choices minimize overhead while enabling page-level autoscaling for serverless LLMs, improved multi-LLM consolidation in clouds and desktops, and a practical first step toward GPU-native OS abstractions that could dramatically raise real-world GPU utilization and lower infrastructure costs.