Amazon SageMaker launches HyperPod Inference Gateway for GPU-aware LLM routing
Amazon announced the SageMaker HyperPod Inference Gateway, a Kubernetes-native EKS addon that routes OpenAI-compatible inference requests using real-time GPU signals (KV cache, queue depth, LoRA residency, etc.) to reduce first-token latency and GPU waste without application changes. The two-tier system offers per-cluster intelligent routing and fleet-wide coordination, deployable via a single InferenceGatewayConfig resource and emitting Prometheus/CloudWatch metrics.