Moe expert offloading on a 2-core Celeron with 2.7GB RAM (github.com)

A recent study explored the performance of mixture-of-experts (MoE) models on a low-resource Intel Celeron N4000 system with just 2.7GB of RAM. It focused on prefetching expert weights from disk to improve model processing times during matrix multiplications using the OLMoE-1B-7B-0924 model. The findings demonstrated that utilizing a prefetching mechanism alongside careful memory management strategies, specifically mlocking essential tensors, yielded a notable increase in token generation rates, achieving approximately 0.114 tokens per second—2.5 times faster than previous designs that relied on more complex caching strategies which ultimately led to inefficiencies and memory over-consumption. This work is significant for the AI/ML community as it addresses the challenge of running advanced models on constrained hardware, facilitating wider accessibility and deployment of powerful neural networks in less favorable environments. The research indicates that, contrary to prior assumptions, effective weight offloading and memory management can enhance performance even in systems with limited computational power, paving the way for future innovations in MoE architecture optimization under practical hardware constraints.