Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

COMPANY · ENTITY #12439

GPTQ

Related event timeline, sources and context from the news index.

EVENT TIMELINE

4

RESEARCH · 1 SOURCE · arXiv cs.AI

Edge-Native Multimodal RAG paper finds NF4 quantization enables RAG on 12–16 GB Turing GPUs

arXiv:2611.14924v1 presents an engineering and empirical study of a multimodal retrieval-augmented generation (RAG) pipeline optimized for consumer Turing GPUs (e.g., NVIDIA Tesla T4). The authors report that GPTQ gives higher throughput (51.6 tok/s) but suffers context instability past ~2048 tokens on Turing hardware, while NormalFloat 4-bit (NF4) preserves long-context stability with ~81 ms/token latency; combining NF4 with a decoupled Florence-2 vision module yields visual reasoning with only ~430 MB VRAM overhead and an overall 8.37 GB footprint, enabling RAG on 12–16 GB cards, and structured few-shot prompting recovers quantization-induced reasoning loss (math accuracy from 35% to 92% over 200 trials) and structured prompting can cut latency by 47%.

6.0

RESEARCH · 1 SOURCE · arXiv cs.AI

MPQ: message-passing approach to post-training quantization (arXiv:2610.32127v1)

This arXiv preprint proposes MPQ (Message-Passing Quantization), which frames row-wise post-training quantization as posterior inference over discrete codebooks and preserves soft posterior information across weight blocks using vector approximate message passing before committing to hard codewords via block coordinate descent. The authors report improved perplexity and zero-shot accuracy versus hard-decision baselines (e.g., GPTQ/LDLQ) on 2-bit lattice and 3–4-bit scalar quantization for dense LLMs from the Llama, Qwen, and Mistral families, and provide a reference implementation on GitHub.

7.0

MODELS · 1 SOURCE · arXiv cs.AI

DRQ: Distributionally Robust refinement for low-bit LLM quantization

The arXiv paper shows that minimizing reconstruction loss in weight-only PTQ can fail or even harm downstream performance when input activation distributions shift, and proposes Distributionally Robust Quantization (DRQ), a post-hoc refinement that minimizes worst-case reconstruction loss over constrained input distributions. DRQ adjusts integer codes within existing quantization grids (keeping quantization parameters and inference unchanged) and reportedly improves models quantized by multiple PTQ methods including AWQ, GPTQ, and ParoQuant across both dense and MoE LLMs.

7.0

RESEARCH · 1 SOURCE · arXiv cs.AI

Measuring the microtask eligibility gap for off-the-shelf SLMs (arXiv:2610.00025v1)

This paper builds a benchmark of four microtasks around LLM planners (auto-approving commands, writing memory, tool selection, ranking turns) with pre-specified, CI-backed eligibility thresholds and tests off-the-shelf small language models (Qwen3 0.6/1.7/4/8B, greedy FP16, no tuning). Across 16 model×task configurations, none met the practitioner-defined thresholds (0/16). The authors introduce a logprob decision-threshold diagnostic separating capability deficits from decoding-threshold failures, show 4-bit quantization methods (RTN/GPTQ/AWQ) generally worsen performance without enabling eligibility, replicate the gap on Llama-3.x (12/12 ineligible), and show results are robust to anchor choice and prompt paraphrases; they recommend placing SLMs behind a baseline that already meets the CI-backed threshold and only using SLMs where that baseline fails.

7.0