NEWS · MODELS · #1757
DRQ: Distributionally Robust refinement for low-bit LLM quantization
The arXiv paper shows that minimizing reconstruction loss in weight-only PTQ can fail or even harm downstream performance when input activation distributions shift, and proposes Distributionally Robust Quantization (DRQ), a post-hoc refinement that minimizes worst-case reconstruction loss over constrained input distributions. DRQ adjusts integer codes within existing quantization grids (keeping quantization parameters and inference unchanged) and reportedly improves models quantized by multiple PTQ methods including AWQ, GPTQ, and ParoQuant across both dense and MoE LLMs.
KEY POINTS
- The arXiv paper shows that minimizing reconstruction loss in weight-only PTQ can fail or even harm downstream performance when input activation distributions shift, and proposes Distributionally Robust Quantization (DRQ), a post-hoc refinement that minimizes worst-case reconstruction loss over constrained input distributions.
- DRQ adjusts integer codes within existing quantization grids (keeping quantization parameters and inference unchanged) and reportedly improves models quantized by multiple PTQ methods including AWQ, GPTQ, and ParoQuant across both dense and MoE LLMs.
- DRQ targets the mismatch between calibration and real input activation distributions to improve downstream performance of low-bit PTQ without adding inference overhead.
WHY IT MATTERS
DRQ targets the mismatch between calibration and real input activation distributions to improve downstream performance of low-bit PTQ without adding inference overhead.