When Equivalent Weights Train Differently (jiha-kim.github.io)

A recent study highlights the significance of coordinate choices in training transformer attention mechanisms, detailing how different parameter representations can lead to varying training outcomes despite yielding equivalent products. This phenomenon arises because optimizers can distinguish between these representations at a coordinate level, revealing a disconnect between the model's forward pass and the optimizer's perspective. The research emphasizes that while the attention head's query-key and value-output products remain unchanged functionally, the optimization process is sensitive to the specific coordinate choice for parameters, potentially affecting convergence and training efficiency. The findings carry substantial implications for the AI/ML community, particularly in refining training techniques for large models. By recognizing that common optimizers do not fully mitigate coordinate dependencies, the study suggests that current methods like SGD, AdamW, and Muon-style updates may not provide complete robustness against dimensional transformations. The proposed solutions—such as periodic opposite-Gram corrections or leveraging random gauge perturbations—aim to improve optimizer efficiency and convergence stability. These insights could inspire new optimization strategies that are more resilient to factor gauge variations, ultimately enhancing the training of transformer architectures and their applications across AI fields.