Amazon SageMaker Inference adds prefix-aware routing to reduce LLM latency
Amazon SageMaker Inference now offers prefix-aware routing, a strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, this reduced P50 time-to-first-token by up to 77% and increased KV cache hit rates from about 25% to over 80%.