Qwen3.8-Flash-Next non-uniform quantization runs on 2 RTX3090s (huggingface.co)

A new development in AI model quantization has emerged with the release of Qwen3.8-Flash-Next, which can run a 180-billion parameter model on just two NVIDIA RTX 3090 GPUs. This model utilizes non-uniform quantization techniques (specifically GSQ and RCO) to optimize performance, achieving a score 2.85 percentage points higher than the benchmark BF16 on the MMLU-Pro benchmark, alongside a slight increase in perplexity. The model innovatively stores embeddings on disk while managing the key-value (KV) cache in RAM, highlighting improved efficiency in resource utilization. This achievement is significant for the AI/ML community as it demonstrates the potential for large language models to operate on more accessible hardware configurations, reducing costs and expanding accessibility for researchers and developers. The implementation of a 3.5-bit GGUF quantization target allows for tailored precision based on tensor sensitivity, paving the way for advanced multimodal applications. By enabling independent reproduction of these quantization methods and making them available for further exploration, this development encourages collaborative research and experimentation within the AI field.