Inferssd – Run models larger than RAM or VRAM by streaming MoE experts from SSD (github.com)

InferSSD has been launched as a groundbreaking tool that allows users to run mixture-of-experts (MoE) models larger than their available RAM or VRAM by streaming model experts directly from SSD storage. This innovation leverages the llama.cpp framework and supports standard GGUF files, offering users features such as browser-based chat and an OpenAI-compatible API. Currently, it supports impressive models like DeepSeek-V4.1-Flash-1017, with a staggering 763 billion parameters, and Muse-Spark-Open, totaling 816 billion parameters, demonstrating the potential for handling massive AI workloads without requiring extensive local hardware resources. This development is significant for the AI/ML community as it addresses one of the key limitations of deep learning—memory constraints—enabling more researchers and practitioners to work with larger and more complex models without the need for expensive hardware. Benchmarks indicate substantial speed improvements over previous frameworks: for instance, InferSSD on an RTX 5090 GPU achieves a decoding speed of 164.5 token/s, which is over 20 times faster than traditional llama.cpp implementations. With support for multiple operating systems and significant hardware configurations, InferSSD paves the way for more flexible and scalable AI applications, ultimately democratizing access to cutting-edge AI technologies.