Amazon SageMaker AI adds built-in concurrency sweeps to right-size generative AI endpoints
Amazon describes using concurrency sweeps—integrated in SageMaker AI Inference Recommendations—to benchmark generative AI endpoints by increasing simultaneous request levels and measuring throughput and latency. The post walks through deploying the NVIDIA Nemotron-3 Nano 30B on an ml.g7e.2xlarge with the vLLM container and running CreateAIBenchmarkJob (with workload defined via CreateAIWorkloadConfig) to discover saturation points and right-size capacity.