Micro-Expert-Router: Running Mixtral-Class Moe Models on NVMe SSDs Without a GPU (github.com)

The recently unveiled Micro-Expert-Router is a Rust execution engine designed to run Mixture-of-Experts (MoE) models on NVMe SSDs without needing a GPU. This innovative architecture enhances the efficiency of large language models by storing the router in RAM and hot-swapping individual expert models from an NVMe drive into RAM as needed. Leveraging the high sequential read speeds of modern PCIe-4/5 NVMe SSDs, the engine can quickly pull expert data required for inference, significantly reducing the I/O cost compared to traditional memory usage. This development is significant for the AI/ML community as it allows for the deployment of larger models on more modest hardware configurations by treating SSDs as the primary weight store while keeping DRAM for active expert caching. The implementation of on-disk quantization and advanced predictive specification mechanisms optimizes the process, enabling interactive token rates for large models, even when full parameter sets exceed DRAM capacities by 10-100 times. By efficiently managing I/O and caching strategies, the Micro-Expert-Router paves the way for cost-effective development and deployment of large-scale AI models, marking a crucial advancement in the field of machine learning.