A traffic spike hits your self-hosted text-to-speech (TTS) endpoint, and requests start queueing. Latency climbs past your service level objective (SLO) right when the most callers are listening. The auto scaling policy behind that endpoint was supposed to prevent this, but the target value it scales on was a guess, and guesses like that fail silently until real traffic proves them wrong.

Get that number wrong, and it’s paid for in one of two ways, and neither shows up as an alarm. Guess too conservative, and the fleet runs GPU instances that sit mostly idle. That cost lands on every invoice but never trips an incident report. Guess too aggressive, and the fleet carries a risk nobody sees. Each instance holds more concurrent requests than it can serve inside its latency promise. Nothing looks wrong until a real traffic spike pushes queueing time past the SLO. That happens right when the most callers are listening. That’s the hidden gap. A policy that looks fine on the console can already be failing the people it’s supposed to serve.

In this post, we deploy Kokoro-82M , an 82-million-parameter TTS model, as a custom container on an Amazon SageMaker AI real-time endpoint. AWS Cloud Development Kit (AWS CDK) stacks handle the deployment. We replace guesswork with two measurements taken against the live endpoint. Q_max is the highest concurrency one instance sustains before breaching the SLO, and T_total is how long a scale-out takes end to end. We derive the scale-out threshold, scale-in threshold, and both cooldowns from those two numbers by arithmetic.

Source link