Mistral Medium 3.5 YaRN bug fix (huggingface.co)

Mistral has announced a significant fix to the Transformers configuration affecting their Medium 3.5 models, resolving a bug that led to performance degradation particularly when handling medium to long contexts. This issue stemmed from incorrect computations in the ROPE (Rotary Positional Encoding) algorithm, which has now been corrected thanks to collaboration with a developer known as Unsloth. While the fix is effective for newly updated models and Unsloth's GGUFs, the bug had been widespread among all GGUFs, quantized versions, and fine-tuned models created with outdated configurations. The significance of this fix lies in its potential to restore expected performance levels across various applications within the AI/ML community, especially for users relying on precise context handling in their models. The update has reportedly led to a noticeable drop in perplexity (PPL), indicating improved model efficiency. Moreover, it's important to note that these issues did not impact the vLLM format, which continues to be the recommended option for users. Mistral is actively working to enhance testing coverage across its ecosystem, emphasizing collaboration and community feedback to further refine its model offerings.