Running 800B-parameter LLMs on consumer hardware with InferSSD (wici.ai)

InferSSD has been announced as an open-source inference runtime designed to enable the execution of massive language models (LLMs) with up to 816 billion parameters on consumer hardware by leveraging NVMe storage. This innovative approach allows users to run models like the DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open, which otherwise would exceed typical memory limits on laptops and desktops. By keeping routed expert weights on SSD instead of RAM, InferSSD efficiently reads the necessary model parameters during inference, allowing for impressive throughput rates. For example, an RTX 5090 setup achieved processing speeds of 164.5 tokens per second with the DeepSeek model and 106.6 tokens per second for the Muse model. This development is significant for the AI/ML community as it democratizes access to high-capacity models, making advanced AI capabilities more widely available without the need for expensive dedicated hardware. InferSSD employs a sophisticated architecture that utilizes caching and shared computations to optimize performance while managing bandwidth constraints associated with SSDs. Its design balances expert weight management, prompt batching, and efficient I/O scheduling, making it a promising solution for developers aiming to work with large-scale AI models in environments with limited memory resources. Overall, InferSSD represents a pivotal advancement in making powerful AI tools usable on everyday consumer devices.