A Fix for Softmax and Cross-Entropy (jonbarron.info)

Recent research has identified and addressed two critical numerical issues in the standard implementations of softmax and cross-entropy functions widely used in AI and machine learning frameworks. The first issue, known as Softmax Collapse, leads to a zero gradient for the correctly predicted target class when its logit significantly exceeds others, disrupting the balance of gradients. The second issue, Softmax Cancellation, occurs during knowledge distillation, where gradients inaccurately remain non-zero even when the logits from the student and teacher networks are equivalent. The authors propose enhanced implementations of log-softmax and cross-entropy that resolve these bugs effectively. These fixes significantly improve the stability and accuracy of training models, particularly those using adaptive optimization algorithms like Adam. By ensuring that gradients sum to zero and accurately reflect the model's predictions, the new implementations mitigate problems associated with loss spikes and training inefficiencies previously observed. This advancement not only enhances the training process but also sets a new standard for implementing softmax operations across various machine learning frameworks, ensuring robustness against numerical instability in gradient computations.