Measured autoscaling for TTS on Amazon SageMaker using Kokoro-82M
A how-to demonstrates deploying Kokoro-82M (an 82M-parameter non-autoregressive TTS) as a custom container on an Amazon SageMaker AI real-time endpoint via AWS CDK and replacing a guessed autoscaling target with one derived from two live measurements: Q_max (max per-instance concurrency inside the SLO) and T_total (end-to-end scale-out time). The deployment uses SageMaker's native ConcurrentRequestsPerModel metric (reported to CloudWatch) and supports both InvokeEndpointWithResponseStream and InvokeEndpointWithBidirectionalStream transports; tooling converts client-measured concurrency into CloudWatch units and the example run reported a 1.0 ratio between measured and published values.