Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU (github.com)

A developer has showcased a high-efficiency inference engine that enables the Llama 3.1 70B model to run on a single RTX 3090 GPU by utilizing a novel NVMe-to-GPU architecture that bypasses the CPU. This system streams model layers directly into GPU memory using PCIe, achieving remarkable performance improvements—up to a 33x speedup compared to traditional memory access methods. The software employs a 3-tier adaptive caching system that dynamically sizes memory resources, making efficient use of VRAM, pinned RAM, and NVMe storage. This development is significant for the AI/ML community as it opens new pathways for running large language models on consumer-grade hardware, drastically reducing latency and enhancing throughput. The implementation allows for the deployment of complex models without extensive infrastructure and costs, thereby democratizing access to cutting-edge AI technologies. Key technical achievements include the seamless data flow from NVMe storage to GPU compute buffers, avoiding CPU involvement, with the system demonstrating capabilities for different quantization formats and advanced layer management strategies, potentially transforming the landscape of LLM deployment.