Huawei releases MACaQ, its internal quantizer for local inference (github.com)

Huawei has unveiled MACaQ, a cutting-edge quantization tool designed for efficient local inference across a variety of model types, including text, images, video, and audio. This model-agnostic quantizer can handle a massive 122 billion-parameter model in just 64 minutes while using a mere 32.5 GB of device memory. Notably, MACaQ achieves lower perplexity levels than existing techniques like MR-GPTQ and DuQuant++ and can match the performance of Llama-3-8B models with significantly reduced quantization time—11 times faster—making it an attractive option for developers. This release is significant for the AI/ML community as it enhances the accessibility and performance of local inference, crucial for applications that require rapid processing with limited resources. MACaQ supports multiple backends, including llama.cpp, vLLM, and PyTorch, and allows for the quantization of various models into compact checkpoints for streamlined generation. Technical aspects, such as support for 2-, 3-, and 4-bit GGUF formats and integration with Ascend inference runtime, position MACaQ as a versatile and essential tool for researchers and developers looking to optimize model performance without sacrificing quality.