What does LoRA specialized Muon optimizer even mean? (zrt.wtf)

Thinking Machines recently introduced Dimuon, an optimizer tailored for LoRA (Low-Rank Adaptation) within their Tinker framework. While the specifics of how Dimuon operates remain vague, it's designed to improve the performance of LoRA, which is a method for efficiently training models by making further adjustments to low-rank factors of weight matrices. This addition follows a series of other LoRA-focused optimizers, including PoLoRA and Riemannion, which illustrate the community's growing emphasis on optimizing the way these factors are handled. Researchers have consistently highlighted that treating the two components of the LoRA adapter as separate entities might not maximize performance, pointing to the need for specialized optimizers like Dimuon. Initial results suggest that Dimuon outperforms traditional optimizers such as AdamW and comparable variants like Muon, especially as the rank increases, indicating that tailored optimizers can yield significant improvements in model training efficiency. Key technical findings point out that conditioning on feature-side statistics appears to be the main advantage of Dimuon, which allows it to maintain consistent performance across different ranks. As such, for those looking to enhance LoRA training, Dimuon emerges as a strong contender, especially when employing appropriate learning rates and decay schedules, emphasizing the importance of optimizing training settings alongside the choice of optimizer.