Deltatensors – store model fine-tunes as compressed weight deltas (github.com)

Deltatensors has introduced a groundbreaking method for efficiently storing fine-tuned neural network models through near-lossless delta compression. Rather than saving multiple versions of a fine-tuned model, users can now keep a single base model along with small delta files (.wdelta) that capture the differences. Tested on the Qwen2.5-0.5B model fine-tuned on WikiText-2, this approach achieved a perplexity difference of only 0.58%, indicating minimal degradation from the original model. The storage savings are significant, reducing total size from 11 GB to just 3.9 GB across 10 fine-tunes, showcasing a 2.8x overall reduction. This innovation is particularly valuable for the AI/ML community, as it simplifies model management and significantly decreases storage requirements without compromising performance. The deltatensors library allows for easy delta creation and reconstruction, enabling users to manage model variations with limited memory usage. The ability to chain multiple .wdelta files also facilitates tracking and reconstructing full fine-tuning histories, adding a layer of flexibility that traditional methods lack. Overall, deltatensors represents a substantial advancement in model storage efficiency while maintaining high fidelity in performance.