Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe (gitlab.com)

Nvidia has announced GreenBoost, an innovative open-source solution that allows users to transparently extend GPU VRAM by utilizing system RAM and NVMe storage. This Linux kernel module and CUDA userspace shim enable running large language models (LLMs) that exceed a GPU's memory without necessitating modifications to the inference software. GreenBoost operates independently alongside standard Nvidia drivers, leveraging Direct Memory Access (DMA) to route overflow memory to system RAM and NVMe via a PCIe 4.0 connection, while maintaining efficient data movement without CPU intervention. This development is significant for the AI/ML community as it addresses the practical constraints of GPU memory in processing large models. By effectively expanding available memory resources, GreenBoost allows researchers and developers to work with models that were previously unharnessable on consumer-grade hardware, thus democratizing access to powerful AI capabilities. Key technical features include a three-tier memory system: RTX VRAM for active compute, DDR4 for cached data, and NVMe for safety overflow, all monitored and managed in real-time. With an installation script that automates setup and tuning for optimal performance across various systems, GreenBoost promises to enhance the efficiency of LLM usage and further stimulate advancements in AI development without necessitating expensive hardware upgrades.