Show HN: SkewAdam – A tiered optimizer that cuts MoE state memory by 97% (github.com)

A new optimizer called SkewAdam has been introduced, achieving a remarkable 97.4% reduction in memory usage for the state of a large 6.78 billion parameter Mixture-of-Experts (MoE) model during training. Traditionally, using AdamW required over 50 GB of optimizer state, which SkewAdam reduces to just 1.29 GB, enabling it to run efficiently on a single 40 GB GPU. This drastic reduction in memory footprint—alongside a peak training memory decrease from 81.4 GB to 31.3 GB—significantly enhances accessibility for researchers and developers working with large-scale models. SkewAdam's design leverages a tiered allocation strategy where it allocates optimizer states differently for various components of the MoE model. By focusing momentum where it is most beneficial (the model's backbone) and minimizing it for the less critical expert components, SkewAdam has demonstrated superior performance, achieving lower validation perplexity than existing optimizers like AdamW and Muon. This tailored approach not only optimizes memory use but also balances performance, potentially setting a new standard for memory-efficient training in the AI/ML community. The optimizer’s enabling of improved training workflows with substantial model sizes highlights an important advancement in computational efficiency for deep learning applications.