Thinking Machines Ships Dimuon Optimizer (tinker-docs.thinkingmachines.ai)

Thinking Machines has announced the release of the Dimuon optimizer for its Tinker platform, designed specifically for LoRA (Low-Rank Adaptation) fine-tuning tasks. This new optimizer complements the existing AdamW and is tailored to improve performance in supervised fine-tuning scenarios, achieving lower loss in fewer steps without compromising step time. Notably, Dimuon’s effective learning rate is significantly higher—typically 10 to 100 times that of AdamW—prompting users to customize their learning rate settings to optimize results based on their specific models and tasks. The introduction of Dimuon is significant for the AI/ML community as it presents a potentially superior option for handling fine-tuning tasks, especially in supervised contexts where it has demonstrated efficiency. The optimizer allows for a flexible learning rate adjustment during training, enhancing adaptability across various model architectures. Users are encouraged to experiment with their learning rates while benefiting from guidelines on learning rate decay and tuning associated with both Dimuon and AdamW, ultimately providing a sophisticated toolset for training more efficient machine learning models.