NEWS · MODELS · #3149
NVIDIA outlines NVFP4, Vera Rubin platform, and five co-design levers to boost AI throughput and KV-cache efficiency
In a blog post, NVIDIA says it is using five co-designed levers — the 4-bit NVFP4 numerical format, wide expert parallelism in TensorRT-LLM, Dynamo (fleet-aware KV-cache routing and the KV Cache Runner), the Vera CPU with Rubin interconnect, and a CMX pod-level storage tier — to free HBM, raise arithmetic intensity, and improve throughput for agentic inference. NVIDIA reports NVFP4 reduces model footprint roughly 3.5x versus FP16 (1.8x versus FP8), NVFP4 KV cache halves cache footprint versus FP8, and claims the Vera Rubin NVL72 can deliver up to 30x the throughput per megawatt of GB300 NVL72 on an agentic benchmark.
KEY POINTS
- In a blog post, NVIDIA says it is using five co-designed levers — the 4-bit NVFP4 numerical format, wide expert parallelism in TensorRT-LLM, Dynamo (fleet-aware KV-cache routing and the KV Cache Runner), the Vera CPU with Rubin interconnect, and a CMX pod-level storage tier — to free HBM, raise arithmetic intensity, and improve throughput for agentic inference.
- NVIDIA reports NVFP4 reduces model footprint roughly 3.5x versus FP16 (1.8x versus FP8), NVFP4 KV cache halves cache footprint versus FP8, and claims the Vera Rubin NVL72 can deliver up to 30x the throughput per megawatt of GB300 NVL72 on an agentic benchmark.
- These co-design changes, if validated, could materially increase efficiency for large‑context and agentic inference by freeing expensive HBM for KV cache, reducing memory copies, and enabling higher concurrency and arithmetic intensity.
WHY IT MATTERS
These co-design changes, if validated, could materially increase efficiency for large‑context and agentic inference by freeing expensive HBM for KV cache, reducing memory copies, and enabling higher concurrency and arithmetic intensity.