The Dual Store
In one line: training reads history in bulk and serving reads one entity now, and no single storage engine is good at both.
Opposite access patterns
| Offline store | Online store | |
|---|---|---|
| Question | What was this feature for these million entities, at these million past moments? | What is this feature for this one entity, right now? |
| Shape | Columnar, append-only, timestamped | Key-value, latest value only |
| Reads | Bulk scans over months | Point lookups, single-digit milliseconds |
| Retention | Every historical value | Only the current one |
| Typical | Warehouse, lakehouse, object storage | Redis, DynamoDB, Cassandra |
| Cost driver | Storage volume | Memory and throughput |
The retention row is the one that explains everything else. The offline store keeps every value a feature has ever had, with the timestamp at which it became true, because training needs to reconstruct the past. The online store keeps one value per entity, because serving only ever asks about now.
That difference makes them genuinely different systems rather than two copies of the same data.
Append-only, with event time
The offline store's schema carries a detail that matters enormously and looks like bookkeeping.
entity_id | feature_value | event_timestamp | created_timestamp ----------|---------------|--------------------|----------------- user_42 | 31 | 2026-03-01 09:00 | 2026-03-01 09:05 user_42 | 34 | 2026-03-08 09:00 | 2026-03-08 09:04 user_42 | 47 | 2026-03-15 09:00 | 2026-03-15 09:06
Two timestamps, not one.
event_timestamp is when the value became true in the world. created_timestamp is when your pipeline learned it. They differ whenever data arrives late — which is always, for anything derived from external systems.
Keeping both is what lets you answer a question a single timestamp cannot: what did we know at that moment, as distinct from what was true at that moment. Training on the second gives the model information it would not have had, which is the leakage from the foundations module in its most subtle form.
Materialisation
The online store is populated from the offline one, or directly from a stream, and this is where the two are kept consistent.
The materialisation lag is the honest weakness. Between a batch job computing a new value and it landing in the online store, serving returns the previous one. For a daily feature that is fine. For anything session-scoped it is fatal, which is why streaming pipelines write to the online store directly rather than going through the offline one.
Two consequences worth stating:
Materialisation freshness needs its own monitor. Not "did the job succeed" but "how old is the newest value any entity has". A job that succeeds while processing nothing is the failure that looks healthy.
The two stores can disagree legitimately. The offline store has a value the online store has not received yet. That is not a bug, and it means a naive consistency check comparing them will alarm constantly — the check has to account for expected lag, which the monitoring lesson takes up.
Backfilling the offline store from the online one
The reverse direction, which surprises people and is genuinely useful.
Some features only exist online — computed by a streaming pipeline from events that were never written to the warehouse, or derived from request-time context that the batch world never sees. Those features have no history, so no model can be trained on them.
Periodically snapshotting the online store into the offline one builds that history going forward. It does not create the past, and it means that six months from now the feature will have six months of history.
The practical implication: a feature that only exists online today is a feature you can train on in six months. Starting the snapshot early is cheap and the delay is otherwise unrecoverable, which makes it one of the higher-leverage things to set up before you need it.
Choosing the online store
The requirements are ordinary, and one is specific to this use.
Low-latency point reads — a few milliseconds at p99, since this sits in the request path.
Batched multi-get — the ranking chapter established that features are fetched for many candidates at once. A store without an efficient multi-key read forces one round trip per candidate, which is the difference between 15ms and 500ms.
High write throughput — streaming features update constantly, and the write path competes with the read path.
TTL support — stale values should expire rather than being served indefinitely, so absence is at least detectable.
That last one is worth a sentence. Without a TTL, a feature whose pipeline silently stopped six weeks ago is still returned, confidently, forever. With one, it eventually becomes missing — and missing is a condition the serving path can detect and handle, which stale is not.
Key takeaway
Training asks for history in bulk and serving asks for one entity now, so the two stores are genuinely different systems: an append-only columnar history keyed on event time, and a latest-value key-value store answering in milliseconds. Keep two timestamps — when a value became true and when you learned it — because reconstructing what was knowable is what prevents late-arriving data from leaking. The materialisation lag between the stores is the design's weak point and needs a freshness monitor measuring value age rather than job success. And snapshot the online store into the offline one, because a feature that exists only online today is one you can train on in six months.
Next: the join that makes historical features correct.