Serving and Missing Values
In one line: the read is simple and there are two ways to get it wrong — one call per entity, and imputing a value for something that was not there.
Batch the read
The single most important property of the serving path.
A ranking request needs features for a hundred candidates. Fetching them one at a time means a hundred round trips.
Per-key: 100 candidates x ~2 ms round trip = 200 ms Batched: one multi-get of 100 keys = ~10 ms
Twenty times the difference, and it is the whole feature budget. An online store without an efficient multi-key read is the wrong store for this job, which is a selection criterion rather than an optimisation.
Two refinements that follow:
Batch across feature groups too. If a candidate needs features from three groups, that is three multi-gets, not three hundred point reads — and ideally one call if the store supports it.
Cap the batch. A single request asking for ten thousand keys can stall the store for everyone. Chunk large fetches and run the chunks in parallel.
What to fetch, and when
The ordering point from the ranking chapter, now as a serving rule.
The user fetch and candidate generation run in parallel because neither depends on the other. The expensive per-candidate fetch happens after the cheap filter. That ordering is free and it is frequently got wrong.
Missing is not zero
The decision that determines whether a degraded store is visible.
A feature can be absent for several reasons, and they are not the same:
| Reason | What it means |
|---|---|
| New entity | No history yet — legitimately unknown |
| Pipeline failure | The value should exist and does not |
| TTL expiry | It existed, went stale, and was evicted |
| Store timeout | It exists and the read did not return in time |
| Genuinely not applicable | This user has never bought from this seller |
The last is a real value: "zero prior purchases" is a fact. The others are ignorance. Collapsing them into the same zero teaches the model that a pipeline failure looks like a user with no history.
The practical shape: pass a null plus an indicator, and train with missingness present. If features are sometimes missing in production, they should sometimes be missing in training too — otherwise the model has never seen the condition it will face during an incident.
Timeouts and partial results
The store is a dependency in the request path, and it will be slow before it is down.
Give the fetch a timeout well inside the budget. If features get 20ms, the call fails at 20ms rather than waiting.
Proceed with partial features. Better to rank with the features that arrived than to fail the request — provided the model handles absence, which is the previous point doing double duty.
Degrade by feature importance. If some features are known to matter more, fetch them in a first, tighter call and treat the rest as best-effort. This makes the degradation graceful rather than arbitrary.
Caching
Item features are the best cache candidate in the system: shared across all users, slow-changing, and read constantly.
| What | Cache? | Why |
|---|---|---|
| Item features | Aggressively | Shared by every user; change slowly |
| User features | Briefly | Per user, but reused across a paging session |
| Pair features | Rarely | Enormous key space, very low hit rate |
| On-demand features | Cheaper to recompute than to look up |
The cache TTL is a freshness decision, and it compounds with the pipeline's own lag. A feature computed hourly and cached for ten minutes can be seventy minutes stale, which is a different number from the one anyone wrote down. Total staleness is pipeline lag plus materialisation lag plus cache TTL, and it is worth computing rather than assuming.
Key takeaway
Batch the read or the feature fetch consumes the entire budget — a hundred point lookups against one multi-get is twenty times the latency, which makes efficient multi-key reads a store selection criterion. Fetch user features in parallel with candidate generation, and per-candidate features only for survivors. Never impute silently: absence has several distinct causes, and collapsing them into zero turns a store outage into an invisible quality regression, so pass a missingness indicator and train with missingness present. And compute total staleness as pipeline lag plus materialisation lag plus cache TTL, which is usually larger than anyone assumed.
Next: the monitoring that catches skew before a model does.