Tech Meridian ← LIVE FEED
RU

RESEARCH · RESEARCH · #529

Inference Engineering Pareto Atlas (arXiv:2609.17863v1) maps cost/quality/latency trade-offs

The paper releases a Pareto atlas that maps cost, quality, and latency trade-offs for LLM inference by measuring 54 configurations of Qwen2.5-7B-Instruct on vLLM 0.12 across L4, A100, and H100 GPUs and calibrating a simulator that reproduces anchored measurements with <1.5% drift. Key findings: 18 of 36 calibrated configurations reach the Pareto frontier; combinations of optimizations outperform many single methods; AWQ 4bit cuts per-token latency to 0.34× but loses ~5.9% GSM8K accuracy; FP8 weights retain ~99.4% accuracy at 0.61–0.65× latency; a naive FP8 KV cache preserves throughput but answers none of 200 GSM8K questions correctly; H100 is best for tight latency while A100 is best for throughput and cost (~$0.106 per million tokens).

KEY POINTS

  1. The paper releases a Pareto atlas that maps cost, quality, and latency trade-offs for LLM inference by measuring 54 configurations of Qwen2.5-7B-Instruct on vLLM 0.12 across L4, A100, and H100 GPUs and calibrating a simulator that reproduces anchored measurements with <1.5% drift.
  2. Key findings: 18 of 36 calibrated configurations reach the Pareto frontier; combinations of optimizations outperform many single methods; AWQ 4bit cuts per-token latency to 0.34× but loses ~5.9% GSM8K accuracy; FP8 weights retain ~99.4% accuracy at 0.61–0.65× latency; a naive FP8 KV cache preserves throughput but answers none of 200 GSM8K questions correctly; H100 is best for tight latency while A100 is best for throughput and cost (~$0.106 per million tokens).
  3. Provides a calibrated, cross‑GPU benchmark and simulator that clarifies which inference optimizations actually dominate cost/quality/latency trade-offs, guiding deployment choices and future optimizer development.

WHY IT MATTERS

Provides a calibrated, cross‑GPU benchmark and simulator that clarifies which inference optimizations actually dominate cost/quality/latency trade-offs, guiding deployment choices and future optimizer development.

SOURCES & TIMELINE

1
01
ARXIV CS.AI RESEARCH
The Inference Engineering Pareto Atlas: Which Optimizations Dominate the Cost, Quality, and Latency Frontier?

arXiv:2609.17863v1 Announce Type: new Abstract: LLM inference optimizations report speedups on different models, GPUs, prompts, and quality metrics, making them hard to compare or combine. We build a cost, quality, and latency Pareto atlas to identify the best configurations for different deployment constraints. Since exhaustive testing is impractical, we measure 54 configurations of Qwen2.5-7B-Instruct running on vL…

↗