Runtime Dynamic Compression of Mixture of Experts [pdf] (timdettmers.com)

A new conference paper presented at ICLR 2026 introduces "runtime dynamic compression" for mixture of experts (MoE) models, developed by Tim Dettmers from Carnegie Mellon University. The innovation addresses the challenge of efficiently deploying increasingly large open-weight language models on consumer hardware, which often struggle with high memory requirements. This method achieves extreme compression levels of 0.4 to 1 bit per weight, significantly surpassing previous dynamic quantization techniques while maintaining comparable model quality, specifically in terms of WikiText perplexity. The significance of this approach lies in its ability to balance expert removal and dynamic quantization, enabling the smooth swapping of experts during inference without increasing memory load. Two key innovations underpin this method: (1) iterative sensitivity probing for precise memory allocation, and (2) dynamic REAP for asynchronous expert retrieval. This leads to high-quality compressed models that fit within practical memory footprints—24 GB for a 125B-class model and 96 GB for a 550B-class model—revolutionizing access to large models for individuals and smaller companies. By open-sourcing the algorithms and implementations in the bits and bytes2 framework, this research strengthens the case for open-source models over proprietary counterparts, potentially broadening accessibility in the AI/ML community.