Amazon SageMaker HyperPod adds model caching to reduce inference cold starts
Amazon SageMaker HyperPod now supports model caching for inference: model weights and container images can be pre-loaded onto cluster nodes' local NVMe storage so pods read from local disk instead of downloading over the network. AWS says this cuts cold starts from tens of minutes to seconds; the announcement explains how it works and how to enable it.