The Latency Contract: TTFT, ITL, and Streaming
In one line: a request that takes six seconds to complete can feel fast or unusable depending entirely on when the first token appears, so one latency number no longer describes the experience.
Two numbers, not one
**Time to first token is the wait before anything appears. It is set by prefill — prompt length and available compute — plus everything upstream: auth, retrieval, queueing.
Inter-token latency** is how fast text flows after that. It is set by decode, which is memory-bandwidth-bound, and by how full the batch is.
The gap between them can be two orders of magnitude. Time to first token might be 400ms; total completion for 500 tokens might be 15 seconds. Reporting only the second describes a system nobody would ship; reporting only the first hides a system that stalls mid-answer.
The reading-speed ceiling
A useful anchor from the estimation chapter: people read at roughly 5 tokens per second. A model generating 30 tokens per second is producing text six times faster than anyone can consume it.
The consequence is a genuine design conclusion: **past a certain point, faster decode buys nothing. If inter-token latency is already comfortably ahead of reading speed, effort spent making decode faster is wasted, and the same effort spent on time to first token or on cost is not.
That is worth saying out loud, because it redirects an optimisation conversation from where teams instinctively go.
Streaming is architecture, not polish
Streaming is what makes the two-number contract possible, and it constrains the whole request path.
| Buffered response | Streamed response | |
|---|---|---|
| User waits | For the whole generation | For the first token only |
| Perceived speed | Total completion time | Time to first token |
| Transport | Ordinary request/response | Server-sent events or websockets |
| Intermediaries | Anything works | Every proxy and CDN hop must not buffer |
| Failure mid-response | Nothing was sent; retry cleanly | Partial output already delivered |
| Cancellation | Little to reclaim | Stopping early frees real GPU work |
The intermediaries row is where implementations actually break. A load balancer, proxy or CDN configured to buffer responses will collect the entire stream and deliver it at once — which silently converts a streaming system back into a buffered one. Time to first token collapses to total latency and nothing errors.
The two rows after it are the ones designs forget, and both deserve their own treatment.
Partial failure has no clean retry
A buffered request that fails sent nothing, so retrying is safe. A stream that fails after 200 tokens has already delivered them.
The options are all imperfect, and naming the trade is the point:
Restart from scratch. The user sees the answer restart, and you pay for the first attempt's tokens again. Simple and visibly jarring.
Continue from where it stopped. Feed the partial output back as context and ask for continuation. Cheaper, and the seam is often visible because the model does not have the same internal state.
Surface the failure and let the user retry.** Honest, and the right answer more often than teams expect — a visible "something went wrong" beats a silently mangled answer.
Cancellation reclaims real capacity
The path most designs omit, and the one with the best return.
Users close tabs, navigate away and abandon long answers regularly. Without a cancellation path, the GPU keeps generating tokens for nobody — occupying a slot in the batch and consuming KV cache that another request needs.
Because continuous batching schedules at every decode step, a cancelled request can free its slot almost immediately — the mechanism is already there, and the missing piece is usually the plumbing from a closed client connection all the way down to the engine.
At meaningful abandonment rates this is one of the cheapest capacity wins available, and it costs no quality at all.
Where the budget goes
Time to first token is not just prefill, and treating it as such hides most of the latency.
Auth and validation ~10 ms
Cache lookup ~10 ms
Retrieval, if the request needs it ~50 ms
Queue wait 0-500+ ms <- usually the largest and most variable
Prefill ~100-500 ms
------------
Time to first token ~200 ms to several seconds
Queue wait is the term that dominates under load and the one people leave out. Under saturation it is not a small addition to prefill — it is the whole number, which is why the next lesson treats the queue as a first-class design object.
Key takeaway
Generation needs two latency numbers: time to first token, which is what users perceive as responsiveness, and inter-token latency, which only has to stay ahead of reading speed at roughly five tokens per second — past which faster decode buys nothing. Streaming is what makes that contract possible and it constrains the whole path, including every intermediary that must not buffer. Design the two things streaming adds: a partial-failure policy, since a delivered stream has no clean retry, and a cancellation path, which frees a batch slot and its KV cache almost immediately.
Next: the queue, which is where most of the latency actually lives.