Amazon SageMaker HyperPod adds model caching to reduce inference cold starts

KEY POINTS

1. Amazon SageMaker HyperPod now supports model caching for inference: model weights and container images can be pre-loaded onto cluster nodes' local NVMe storage so pods read from local disk instead of downloading over the network.
2. AWS says this cuts cold starts from tens of minutes to seconds; the announcement explains how it works and how to enable it.
3. Model caching reduces inference cold-start latency and variability in large SageMaker deployments, improving response times and operational predictability.