Serving and Freshness
In one line: put every piece of work on the slowest timescale that still meets its freshness requirement, and precompute everything else.
Three timescales
Nothing here should be decided per-component in isolation. There are three places work can happen, and each piece of the system belongs on exactly one.
Batch is for anything whose input changes slowly. Item embeddings over a ten-million-item catalogue are a batch job — the item tower runs over the whole catalogue and writes to the ANN index.
Near-line is the tier most designs omit, and it is where the interesting work lives. Triggered by events rather than by a schedule, running in seconds to minutes. A new item's content embedding is computed on upload, not on the next nightly run — which is what makes the one-hour item-freshness requirement achievable without touching the batch cadence.
Request time is only what genuinely cannot be precomputed: this user, this context, this moment.
The design question for each piece is always the same. What is the freshest input it depends on? That answers which tier it belongs to, and nothing else does.
The user embedding decision
A good example of the trade, and a common follow-up.
Precompute the user embedding in batch and the ANN lookup is a single fast query — but the embedding does not reflect anything the user did today.
Compute it at request time from recent history and it is perfectly fresh, at the cost of a model call inside the budget.
The usual answer is a hybrid: a precomputed long-term embedding combined at request time with a cheap function of the session's last few actions. Long-term taste changes slowly and is expensive to compute; session intent changes in seconds and is cheap. Splitting them by how fast they change is the whole idea, and it is the same principle as the three timescales applied inside one component.
Where the budget goes
Restating with the specifics, because this is where candidates lose time they did not know they were spending.
| Work | Where it can go |
|---|---|
| Item embeddings | Batch, plus near-line for new items |
| User long-term embedding | Batch |
| Session signals | Request time — but as feature lookup, not computation |
| Candidate retrieval | Request time, ANN |
| Feature fetch | Request time, and batched |
| Ranking | Request time |
| Policy and diversity | Request time, on a shortlist |
Feature fetch is the one to plan around. A hundred candidates times many features, done as one lookup per candidate, is the single most common way a recommender misses its budget — a hundred round trips at two milliseconds each is two hundred milliseconds, against a batched multi-get at around ten.
Precomputing the whole slate
For some surfaces the entire recommendation can be computed ahead of time. Email digests, notifications, and the first screen of a feed for a returning user are all candidates.
The trade is straightforward: precomputing is far cheaper per impression and the result is stale by however long ago it ran. It also costs storage for every user, including the large fraction who will not visit.
A useful hybrid, and one worth proposing: precompute the first screen for active users, and serve the rest live. It covers the latency-critical first impression — the one that decides whether the session continues — while keeping the tail cheap.
Degradation
A feed that fails to render is much worse than a feed with mediocre recommendations, so the fallbacks should be designed rather than discovered.
Each rung is worse and each one renders. The bottom rung should be a cached static list that requires no live dependency at all, and it should be exercised regularly rather than trusted — a fallback path that has never run in production is a hypothesis.
Two details that matter. Degradation should be per-component, so a feature-store timeout does not take down ranking. And it must be visible: a system silently serving trending content to everyone looks healthy on latency and error rate while the product is quietly broken. Emit which rung served each request, and alarm on the mix.
Caching, and its one hazard
Recommendations are personalised, so the naive cache key is the user — which gives a hit rate close to zero on a feed the user scrolls once.
What caches well is the layer below: candidate sets change more slowly than rankings, trending lists are shared across all users, and item features are shared across all requests. Cache those and the expensive shared work is amortised while the personalised part stays live.
The hazard is the one every personalised system has: a cache key that omits the user identity serves one person's recommendations to another. It is rare, it is severe, and it is worth stating the rule — any cache holding personalised output keys on identity, and anything that does not is by construction not personalised.
Key takeaway
Put each piece of work on the slowest timescale its freshness requirement allows, and remember the near-line tier — event-triggered work in seconds is what makes new items recommendable within the hour without changing the batch cadence. Batch the feature fetch, because a hundred per-candidate round trips is how the budget is usually lost. And design the degradation ladder down to a static list that always renders, then emit which rung served the request.
Next: the whole thing, as an interview.