MACaQ introduces a fast, model-agnostic quantization method for low-precision weights and activations, reducing costs while preserving accuracy.

A new paper titled "MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations" has been published on arXiv by researchers Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, and Jiawei Zhuang. The study introduces MACaQ, a novel post-training quantization method designed to improve the efficiency of neural network inference by reducing memory and computational costs while maintaining accuracy.

MACaQ is an MXFP4 W4A4 post-training quantization method that operates independently on each linear layer of a neural network. It employs per-channel scaling and block-Hadamard transforms to process activations online, while inverse transforms are absorbed into the weights. This approach eliminates cross-layer dependencies, enabling a simple streaming implementation where only one layer resides in memory at a time.

To enhance quantization accuracy, MACaQ introduces Flip-Refinement, a Hessian-aware coordinate-descent procedure that adjusts FP4 codes to adjacent representable values. Despite the additional activation transforms during inference, MACaQ adds only 2.3% end-to-end overhead compared to pure MXFP4 on the Qwen3-32B model in vLLM.

The MACaQ implementation is available on GitHub at https://github.com/huawei-csl/MACaQ. The repository supports multiple backends, including llama.cpp, vLLM, Diffusers, and PyTorch, allowing users to quantize and run models across different domains. Additionally, MACaQ-R is provided for refining existing GGUF models to improve quality and adapt them to specific workloads while maintaining compatibility with standard llama.cpp.

MACaQ's model-agnostic design and ability to operate without architecture-specific modifications make it a versatile tool for optimizing neural network inference.

Source link