First DeepSeek V4 Flash-Base-Int4 Quant (huggingface.co)

The AI community has marked a notable advancement with the release of the DeepSeek V4 Flash-Base-Int4, which features a groundbreaking INT4 packed-storage quantization for a robust 284 billion-parameter Mixture-of-Experts model. This release not only achieves an impressive mean score of 88% on the Massive Multitask Language Understanding (MMLU) benchmark but also significantly reduces the on-disk size by approximately 45%, dropping from 283 GiB to 156.6 GiB compared to the FP8 baseline. Such efficiency in storage coupled with high performance offers exciting implications for deploying large language models in environments with limited resources. Key technical innovations include the use of non-standard quantization methods and the integration of hybrid types for different model components, such as keeping attention components at FP8 for critical quality retention. The model exhibits bit-exact reproducibility, ensuring consistent outputs across different environments. With a sophisticated loading framework accommodating both INT4 and FP8 modes, developers can opt for speed or memory efficiency as required. This release exemplifies the ongoing evolution of quantization techniques, promising enhanced performance and accessibility for the AI/ML community.