Quantizing Small Models to 3 and 2 Bits with ORA-QAT (www.oracomputing.com)

A research team has announced a breakthrough in model quantization, successfully reducing the Qwen3-4B model's weight quantization from 16 bits to 3 and 2 bits using their proprietary approach, ORA-QAT. This method enables the model to occupy only 22% and 18% of the original BF16 size, while remarkably maintaining 96.5% of the full-precision quality at 3 bits and achieving a significant improvement over standard GPTQ at both levels of quantization. This achievement allows for efficient deployment on consumer GPUs and edge devices, making it a game-changer for resource-constrained environments. Significantly, the study addresses the challenges of quality degradation typically associated with low-bit quantization, especially for smaller models. By integrating quantization-aware training and knowledge distillation, they ensured the model remained robust against quantization errors. Furthermore, they tackled the resource efficiency needed during training by employing a layered packing strategy, allowing the quantized model to be trained effectively on a single GPU with limited memory. The results showcase that while performance at 2 bits does suffer, ORA-QAT facilitates substantial gains compared to traditional methods, making smaller models viable for a broader range of applications in the AI/ML community.