Serving Within the Budget
In one line: end-to-end latency is a few hundred milliseconds at p99 and the fine-grained ranker gets tens of them, so the budget is allocated rather than discovered.
Where the time goes
Network, TLS, routing 40 ms
Auth and request parsing 10 ms
User + context feature fetch 20 ms
Candidate generation 10 ms
Pre-ranking 5 ms
Item + cross feature fetch 15 ms
Ranking 20 ms
Re-ranking and business rules 5 ms
Serialisation and render 15 ms
Headroom for p99 60 ms
------
Total 200 ms
Two observations that redirect effort.
The model is a minority of the budget. Ranking plus pre-ranking is 25ms of 200. A team optimising the model for latency is optimising an eighth of the number.
Feature fetch is 35ms — more than both models combined. Which is the estimation chapter's point arriving again: the online feature store is latency-critical infrastructure, and batching its reads is the highest-value optimisation in the request path.
The parallelism that is free
Several stages have no data dependency on each other, and running them sequentially is a common and avoidable loss.
Three rules fall out.
User features do not depend on candidates. Start that fetch the instant the request arrives, in parallel with candidate generation.
Candidate sources are independent. Fan out to all of them at once with a per-source deadline, and proceed with whatever returned — the degradation from the candidate lesson.
Fetch item features only for survivors. Fetching for ten thousand candidates when the pre-ranker will keep a hundred wastes 99% of the work. Pre-rank on cheap features, then fetch the expensive ones for what survived.
That last one is a real reason the pre-ranker earns its place beyond compute: it also cuts the feature-fetch fan-out by two orders of magnitude.
Caching a personalised response
Full-response caching mostly does not work here — the whole point is that every user sees something different. What does cache:
| Layer | Cacheable? | Why |
|---|---|---|
| Item features | Yes, aggressively | Shared across all users, change slowly — the best hit rate in the system |
| Item embeddings | Precomputed and immutable until the model changes | |
| Candidate sets for cold users | A user with no history gets the popularity list — identical for many | |
| User features | Briefly | Per user; useful across a paging session |
| The final ranked list | Rarely | Personalised and context-dependent — but see paging |
The paging case is the exception worth naming. A user asking for page two should not re-run the whole cascade — cache the ranked list for a short window keyed on the request, and serve subsequent pages from it. It also makes paging consistent, which re-ranking otherwise breaks: without it, an item on page one can reappear on page two because the list was recomputed against slightly different state.
Degrading a cascade
The interesting failure is a stage being slow rather than down, and each stage has a different sensible response.
The third branch is the elegant one, and it is available only because the cascade exists. When the ranker is unavailable, the pre-ranker has already produced an ordering over the same candidates. It is worse and it is free — a fallback that required no extra infrastructure because the architecture already computed it.
The second branch carries the caveat from the monitoring lesson: serving with missing features is only safe if the model handles absence explicitly. If it imputes a zero, a degraded feature store becomes a silent quality regression rather than a visible one.
Personalisation has a floor
A useful sizing thought. If the cascade cannot complete in budget, the honest options are ranked:
Fewer candidates. Halve the candidate count. Recall drops a little; everything downstream gets faster.
Skip the ranker, keep the pre-ranker. A large latency saving for a moderate quality loss.
Skip personalisation entirely. Serve the popularity list. Much worse, and it always works.
Naming that ladder — and noting that the last rung is a cached list depending on nothing in the request path — is the same degradation discipline as the serving chapter, applied to a cascade.
Key takeaway
The models are a minority of the latency budget and feature fetch costs more than both combined, so batching feature reads beats optimising the model. Exploit the free parallelism: user features and candidate sources have no mutual dependency, and item features should be fetched only for pre-ranker survivors, which cuts the fan-out by two orders of magnitude. Cache item features aggressively and the ranked list briefly for paging, which also stops items reappearing across pages. And when the ranker is degraded, fall back to the pre-ranker's ordering — a free fallback that exists only because the cascade already computed it.
Next: measuring a pipeline where every stage caps the next.