Scale AI inference: find the bottleneck before adding capacity
Separate failed requests from slow ones, measure where the delay occurs and control retries.
Add capacity when a repeatable load test shows it is the limiting factor.
The short version
A traffic peak can expose several different problems: requests waiting for capacity, long prompts, slow generation or a deployment restarting after a quiet period.
We’ll use a hypothetical support assistant that receives 8 requests per second in the morning and 45 at its afternoon peak.
- Diagnose01Errors vs slow responsesAn error that returns quickly can make an average response time look better while users get worse results.
- 02Where requests lose timeEach pattern has its own cause: a slow first token, a slow whole answer, a slow first request after a pause, or capacity errors.
- Bound retries03Bound the retriesRetries add load exactly when the service is failing, so cap them and retry only temporary failures.
- Size capacity04Capacity and restartsPlan from what one replica measurably handles, and test the pause after one hour without requests.
- 05Repeat the load testOne smooth run does not cover the peak, a sudden increase and the first request after a quiet period.
