RELEASE · COMPANIES · #657
NVIDIA releases AIPerf, a multiprocess LLM benchmarking tool
NVIDIA AIPerf replaces GenAI-Perf with a ground-up multiprocess architecture designed to avoid client-side bottlenecks for high-concurrency LLM inference benchmarking. It supports 15+ endpoint types, public datasets and trace replay formats (ShareGPT, Mooncake, Baseten, WEKA/AgentX), configurable arrival patterns, and reports TTFT, ITL, latency percentiles and GPU telemetry when DCGM or pynvml are available.
KEY POINTS
- NVIDIA AIPerf replaces GenAI-Perf with a ground-up multiprocess architecture designed to avoid client-side bottlenecks for high-concurrency LLM inference benchmarking.
- It supports 15+ endpoint types, public datasets and trace replay formats (ShareGPT, Mooncake, Baseten, WEKA/AgentX), configurable arrival patterns, and reports TTFT, ITL, latency percentiles and GPU telemetry when DCGM or pynvml are available.
- AIPerf improves the fidelity of LLM inference benchmarks by removing client-side GIL/ single-process limits and providing workload replay and telemetry useful for production-scale measurement.
WHY IT MATTERS
AIPerf improves the fidelity of LLM inference benchmarks by removing client-side GIL/ single-process limits and providing workload replay and telemetry useful for production-scale measurement.