Latency, Tokens, and Streaming

This lesson goes beyond the design

The chapter states a 2–3 second latency requirement, assumes a 4 KB response, and lists caching and cost-aware routing as the techniques that meet it. It never computes how long generating 4 KB of text actually takes, and it never mentions streaming — which is how every production chatbot meets a latency budget.

This lesson works the arithmetic and covers time-to-first-token, the prefill/decode split, and batching. These are the concepts an interviewer is probing for when they ask how you would make an LLM service feel fast.

The budget does not survive contact with the arithmetic

Low latency: Responses should arrive within two to three seconds to maintain a natural conversational feel.

Average response size: ~4 KB (LLM-generated text is usually longer than the input).

A 4 KB response takes five to twenty seconds to generate, not two to three

Token generation is sequential. The model produces one token, feeds it back in, produces the next. There is no parallelizing within a single response — each token depends on the one before it.

So response latency is not a fixed cost; it is proportional to output length.

4 KB response ~= 4,000 characters ~= 1,000 tokens (roughly 4 chars/token)

At  30 tokens/s  ->  33 seconds
At  50 tokens/s  ->  20 seconds
At 100 tokens/s  ->  10 seconds
At 200 tokens/s  ->   5 seconds     <- fast, large-model serving

Even at an aggressive 200 tokens per second, a 1,000-token answer takes five seconds — and that is before retrieval, before the network, before moderation.

The chapter's stated techniques do not close this. Caching helps only on a hit; a cache miss still generates. Cost-aware routing to a smaller model helps — smaller models generate faster — but not by the factor required, and the queries routed to the big model are precisely the ones with long answers.

So the requirement and the assumption are inconsistent: you cannot deliver a 1,000-token response in 2–3 seconds by making generation faster. You have to change what "arrive" means.

In LLM serving, latency is a function of output length, so any budget stated without an output-length assumption is incomplete.

The resolution: streaming, and time-to-first-token

Streaming changes the metric from time-to-completion to time-to-first-token

The insight is that the user does not need the whole answer to start reading it. A human reads at roughly 4–5 words per second; a model generating at 50 tokens per second is already producing text faster than the user consumes it.

So the perceived latency is not when generation finishes — it is when text starts appearing.

TTFT  (time to first token):   what the user experiences as "the wait"
                               -> typically 200-500 ms
TPOT  (time per output token): whether it feels smooth once started
                               -> needs to beat reading speed, ~20-30 tok/s
Total generation time:         mostly IRRELEVANT to perceived latency

This is why the 2–3 second requirement is achievable after all — as a TTFT budget rather than a completion budget. And it reframes the optimization target entirely:

BLOCKING mindset: make generation FASTER      -> hard, expensive, bounded
STREAMING mindset: make the FIRST token fast  -> tractable

Streaming converts a throughput problem into a latency-to-first-byte problem, and the second is far easier to solve.

It is the same instinct as progressive image rendering and HTTP chunked transfer: deliver partial results as they become available rather than buffering to completion. What makes it unusually effective here is that the output is text a human reads sequentially — so partial results are not a degraded experience, they are the natural one.

Prefill and decode

Inference has two phases with completely different performance characteristics

Understanding why TTFT is separately optimizable requires splitting generation in two.

Prefill — process the entire input prompt and produce the first token. All input tokens are processed in parallel, so this phase is compute-bound and fast per token, but it scales with prompt length.

Decode — generate each subsequent token, one at a time. Each step is a full forward pass producing a single token, so this phase is memory-bandwidth-bound and scales with output length.

PREFILL:  parallel over the prompt      -> determines TTFT
DECODE:   sequential, one token at a time -> determines TPOT

Two consequences that matter for this design:

Long prompts hurt TTFT specifically. And this system's prompts are long by construction — RAG injects retrieved chunks, and the session cache injects conversation history:

System instructions      ~200 tokens
Retrieved chunks (top-k) ~2,000 tokens     <- RAG
Conversation history     ~1,000 tokens     <- multi-turn
User query                  ~30 tokens
                          ------------
Prompt                   ~3,200 tokens, for a 30-token question

The features that make the bot good — grounding and memory — are the features that make it slow to start. That is a real tension, and it is why Lesson 5 cares about how many chunks to retrieve and Lesson 8 cares about truncating history.

Long outputs hurt total time, which streaming makes tolerable. So the two phases map cleanly onto the two mitigations: shorten the prompt to improve TTFT; stream the output to hide decode.

Batching

Batching raises throughput and costs latency — and continuous batching mostly removes the trade

The chapter lists "batched inference" under cost efficiency without explaining the mechanism or its cost.

GPUs are efficient at large matrix operations and wasteful on small ones. Serving one request at a time leaves the hardware badly underutilized, so inference servers batch multiple requests into one forward pass:

Batch of 1:   GPU mostly idle between operations -> terrible throughput
Batch of 32:  near-full utilization              -> ~10-20x the throughput

The naive cost is latency: static batching waits to accumulate a batch, and a request that arrives just after a batch departs waits for the next one. Worse, all requests in a batch finish together, so a short answer waits for the longest answer in its batch.

Continuous (in-flight) batching fixes both. Because decode is token-by-token, the server can add a new request at any token boundary and evict a finished one immediately:

STATIC:     wait for N requests -> run to completion -> all finish together
CONTINUOUS: maintain a running batch; join and leave at token boundaries
            -> no waiting to start, no waiting for the slowest peer

That is why real serving stacks quote throughput gains with minimal latency penalty, and it is the single largest lever on the GPU tier's effective capacity — which Lesson 2 identified as the system's actual bottleneck.

Batching is what makes the 200-requests-per-second GPU figure achievable at all, and continuous batching is what makes it achievable without sacrificing TTFT.

The full latency budget

Moderation is placed after generation, which breaks streaming

The design says the content moderation service "scans LLM outputs for policy violations, PII leakage, or harmful content before delivery to the user."

That is correct as a safety posture and it is incompatible with streaming as usually implemented, because you cannot scan a complete response before delivery if you are delivering it as it is generated.

MODERATE-THEN-SEND:  safe, but reintroduces the full 10-second wait
STREAM-THEN-MODERATE: fast, but unsafe content reaches the user first

The production resolutions, in increasing order of sophistication:

Stream with buffered moderation — hold a small rolling window (a sentence or two) behind the moderation check, so you deliver slightly behind the generation front. Adds a few hundred milliseconds rather than seconds.

Moderate the prompt as well as the output, so a large share of problems never reach generation.

Abort mid-stream. If moderation flags content partway through, stop generating and replace the message. The user sees a truncated answer replaced by a fallback — visibly imperfect, and far better than either extreme.

A post-generation safety gate and a streaming delivery model are in tension, and the standard resolution is to moderate a sliding window rather than the whole response. The design states the gate and never confronts the tension, because it never introduces streaming.

How to answer 'how do you meet a 2-3 second latency budget?'

The weak answer is the chapter's: caching and routing to smaller models. Both help and neither is the mechanism.

The strong answer:

"First I'd challenge the metric. A 1,000-token answer takes five to twenty seconds to generate depending on the model — token generation is sequential, so latency scales with output length, and no amount of caching changes that on a miss.

So the budget has to be time-to-first-token, not time-to-completion, and the mechanism is streaming. Users read at four to five words a second; a model generating at fifty tokens a second is already outrunning them. What they experience as 'the wait' is TTFT, and 200 to 500 milliseconds is achievable.

That reframes the optimization. TTFT is dominated by prefill, which scales with prompt length — and this system's prompts are long by construction, because RAG injects retrieved chunks and multi-turn history injects the conversation. So the levers are retrieving fewer chunks, summarizing older turns, and caching, not making decode faster.

On the GPU tier, continuous batching is the single biggest throughput lever — it lets requests join and leave the running batch at token boundaries, so you get the utilization of batching without the queueing penalty.

One tension I'd flag: the design moderates the complete output before delivery, which is incompatible with streaming. I'd moderate a sliding window a sentence or two behind the generation front, and abort mid-stream if something is flagged."

Naming TTFT, prefill-versus-decode, and continuous batching is the clearest signal that you have served a model rather than called an API.

Next: the APIs and the architecture.