Tech Meridian ← ENTITY INDEX
PROMY MERIDIAN RU

COMPANY · ENTITY #34088

NF4 (NormalFloat 4-bit)

Related event timeline, sources and context from the news index.

EVENT TIMELINE

1

RESEARCH · 1 SOURCE · arXiv cs.AI

Edge-Native Multimodal RAG paper finds NF4 quantization enables RAG on 12–16 GB Turing GPUs

arXiv:2611.14924v1 presents an engineering and empirical study of a multimodal retrieval-augmented generation (RAG) pipeline optimized for consumer Turing GPUs (e.g., NVIDIA Tesla T4). The authors report that GPTQ gives higher throughput (51.6 tok/s) but suffers context instability past ~2048 tokens on Turing hardware, while NormalFloat 4-bit (NF4) preserves long-context stability with ~81 ms/token latency; combining NF4 with a decoupled Florence-2 vision module yields visual reasoning with only ~430 MB VRAM overhead and an overall 8.37 GB footprint, enabling RAG on 12–16 GB cards, and structured few-shot prompting recovers quantization-induced reasoning loss (math accuracy from 35% to 92% over 200 trials) and structured prompting can cut latency by 47%.

6.0