Latency and Cost Pick the Model

In one line: the most accurate model you can train is usually not the model you can serve, and the budget that rules it out is arithmetic you should do out loud.

Do the arithmetic before choosing

A ranking request has a latency budget — say 200 milliseconds end to end, of which the model gets perhaps 30. A catalogue has 100 million items. A good ranking model takes roughly a millisecond per item.

100,000,000 items  ×  1 ms per item  =  100,000 seconds per request
Budget:                                       0.03 seconds
Gap:                                    ~6 orders of magnitude

That is not a tuning problem. No amount of hardware closes six orders of magnitude, so the architecture has to change — and the arithmetic is what proves it rather than asserts it.

This is the single most useful habit in the round. Derive the constraint numerically, then let it choose the design. A candidate who says "we'd use two-stage retrieval" has recalled a pattern. A candidate who shows the budget is violated by six orders of magnitude and then derives two-stage retrieval has demonstrated the reasoning that produced the pattern — and can therefore adapt it when the numbers are different.

Budget the whole request, not just the model

The model's share is what is left after everything else. Writing the budget down as a table forces that:

Two things fall out. The feature fetch is often larger than people expect and is a genuine design constraint — it is why online feature stores are latency-critical infrastructure. And the headroom line matters: designing to the p50 budget guarantees you miss the p99 target, because the tail is where the queueing lives.

The cascade that follows

Once the budget rules out scoring everything with a good model, the shape is forced: use a cheap model to shrink the candidate set, then spend the expensive model on what survives.

The stages have genuinely different jobs, and the point most worth making is that they optimise different metrics.

Retrieval is judged on recall. An item it drops can never be recovered, because ranking only ever sees what retrieval returned. Ranking is judged on precision and ordering over the set it was handed. Tuning retrieval for precision is a common and costly error: it throws away items the ranker would have loved, and the loss is invisible because you never see what you did not retrieve.

The same shape, many names

It is one pattern wearing different clothes, and recognising that is worth more than memorising each instance.

The moderation row generalises the pattern in a useful direction: the expensive stage does not have to be a model. Routing the uncertain cases to people is the same architecture, and it is usually the right answer when errors are costly and volume is survivable.

The levers

When the budget is tight, these are the moves, roughly ordered by how much adopting them costs you.

The last row is the trade people state backwards. Batching improves cost and throughput by making each request wait for company. It is a throughput lever that spends latency, and offering it as a latency optimisation is a tell that the candidate has not operated one.

Precomputation is the strongest lever, where it applies

Two-tower retrieval is the clean example. Item embeddings do not depend on who is asking, so they are computed offline and indexed. At request time only the user tower runs, and the rest is a nearest-neighbour lookup against a prebuilt index.

That is how sub-millisecond retrieval over hundreds of millions of items is possible at all.

The insight generalises past recommendation: find the half of the computation that does not depend on the request, and move it out of the request. The cost is that the two towers cannot see each other — no feature can combine user and item, which is precisely why a ranking stage with cross features exists downstream.

Cost is a first-class constraint

For a classic model, serving cost is usually small next to engineering cost, and candidates are used to ignoring it. For a large generative model it can dominate the entire product economics, and a design that ignores it is not a design.

Get into the habit of stating cost per request alongside latency:

1,000,000 conversations / month
  × 8 turns each
  × ~1,500 tokens per turn
  = ~12 billion tokens / month

Whether that is affordable depends entirely on the per-token price and the model tier, and that is the calculation which decides whether the product exists. It is also what motivates most of the LLM-specific architecture in this course — routing cheap requests to small models, caching semantically similar queries, and capping context length.

Next: what happens to all of this over time.