Magazine/Scale inference/GUIDE · 5 STEPS

Scale AI inference: find the bottleneck before adding capacity

Separate failed requests from slow ones, measure where the delay occurs and control retries.

Add capacity when a repeatable load test shows it is the limiting factor.

Step 01 / 05 · Diagnose

Separate errors from slow responses

An error that returns quickly can make an average response time look better while users get worse results.

  • Count failed requests separately from successful ones.
  • Look at the slow end of the distribution. p95 is the time within which 95% of requests finish; p99 covers 99%.
  • Use the same measure consistently, such as time to first token or total response time.
  • Compare normal traffic with the peak. Check whether the problem is more errors, longer waits or both.
Before moving on: identify which measure fails and under what traffic conditions.
Exhibit 2Log the status next to both timings, so fast errors cannot pass as fast answersrequest.log · for each request, record
  • Status code and error message
  • Time to first token until the first response token arrives
  • Time to complete answer until the complete answer arrives
  • Input and output size
  • Trace ID and timestamp of the request
Source Lyceum
Step 02 / 05 · Diagnose

Find where each request loses time

Each pattern has its own cause: a slow first token, a slow whole answer, a slow first request after a pause, or capacity errors.

  • Split response time into the wait for the first token and the time spent generating the rest of the answer.
  • Streaming lets users see output as it arrives. It does not make the model produce the first token sooner.
  • Check client and stream-inactivity timeouts against measured behaviour. Increasing a timeout can prevent premature cancellation, but it does not remove the underlying delay.
  • Use the status page alongside your own measurements. A provider’s short test requests do not represent every customer workload.
Before moving on: have a likely cause supported by request timings or error evidence.
Exhibit 3A slow first token points to queueing and prompts; a slow whole response to output and model speed
SymptomWhat to investigate
First token is slow, generation is steadyQueueing, prompt length and prompt processing.
The whole response is slowOutput length, model speed, client and network behaviour.
Only the first request after a quiet period is slowA paused deployment starting again.
Requests receive capacity errorsAvailable capacity, traffic limits and retry load.
Source Lyceum. Diagnostic guide, not measured data.
Step 03 / 05 · Bound retries

Keep retries from making the peak worse

Retries add load exactly when the service is failing, so cap them and retry only temporary failures.

  • Every retry adds another request. If all 45 requests per second fail and each is retried three times, the service can receive 180 attempts per second.
  • Limit both the number of attempts and the total time spent trying. Include each attempt’s timeout in the overall budget.
  • Use our serverless documentation to map specific error responses to the appropriate action. Log the trace ID when a request fails.
Before moving on: show that retries stop within the intended budget and do not multiply an outage into a larger traffic spike.
Exhibit 4Only temporary failures earn a retry; invalid requests need a fix, generation limits a diagnosis
temporary failure
Retry
  • Increase the wait after each attempt
  • Add random variation, called jitter
  • Honour Retry-After when the server supplies one
invalid request
Fix, don’t resend
  • Unsupported parameter
  • Input that exceeds the model’s limit
generation-limit error
Diagnose
  • Streaming does not bypass a server-side generation limit
  • A shorter request or different workflow may be necessary
Source Lyceum; serverless documentation
Step 04 / 05 · Size capacity

Check capacity settings and restart delays

Plan from what one replica measurably handles, and test the pause after one hour without requests.

  • A dedicated endpoint can add replicas, which are copies of the model server, within configured limits. Scaling targets guide when to add capacity; they are not promises of throughput or latency.
  • Measure how much traffic one replica can handle while meeting your response-time requirement. Use that measurement for planning.
  • Our dedicated deployments scale to zero after one hour without requests. The next request resumes the deployment and waits while replicas start.
  • Setting min_replicas to one does not disable that pause. If first-request latency matters, confirm the supported operating arrangement before launch.
Before moving on: know your replica limits, measured capacity and behaviour after idle periods.
Exhibit 5At an assumed 10 requests/s per replica, a 45 requests/s peak needs 5 replicasPlanning capacity, requests per second
1 replica
10 req/s
2 replicas
20 req/s
3 replicas
30 req/s
4 replicas
40 req/s
5 replicas
50 req/s
Source Example capacity estimate, not a guarantee. Hypothetical steady traffic and equal-sized requests; the per-replica rate is an assumption, not a measured capacity. Excludes startup time and variability: real requests vary, replicas take time to start and peaks need headroom.
Step 05 / 05 · Size capacity

Repeat the load test before choosing capacity

One smooth run does not cover the peak, a sudden increase and the first request after a quiet period.

  • Write the target before testing. Keep request sizes, output lengths and traffic patterns comparable before and after a change.
  • Measure errors separately and include the time needed to add replicas.
  • Then compare serverless and dedicated options using the measured workload and a current quote. Include response time, reliability requirements, quiet hours and operational effort alongside cost.
  • If capacity errors continue, send us the trace IDs, UTC timestamps and a description of the load. That gives us specific requests to investigate.
Before expanding traffic: confirm the setup meets the target under repeatable load, with bounded retries and a way to recover.
Exhibit 6Write the target down first, then test all three traffic shapes against itloadtest.target · hypothetical support assistant
first tokenwithin two seconds
requests95%
requests / second45
traffic shapespeak · sudden increase · first request after a quiet period
Source Illustration: an example target for the hypothetical support assistant.

Your next workload starts here