The Feature Store
In one line: this is the only layer in the design that exists because of machine learning. Everything distinctive about the chapter is here.
The architecture
The feature registry serves as a central catalog, maintaining metadata such as schemas, owners, and version history, allowing teams to discover and reuse features.
The offline store stores complete historical feature values, enabling point-in-time accurate training data generation and time-travel queries.
The online store is a low-latency key-value database providing immediate access to the most recent feature values.
A single feature computation pipeline writes to both stores using the same logic. This prevents training-serving skew.
Why two stores
| Offline store | Online store | |
|---|---|---|
| Contains | Complete history — every value at every time | Latest value only |
| Query | Millions of rows, few columns, as-of a past timestamp | One entity, all its features, now |
| Latency | Minutes — nobody waits | Milliseconds — a user waits |
| Access shape | Scan — columnar, analytical | Point lookup — key-value |
| Technology | BigQuery, Parquet, Delta | Redis, DynamoDB, Cassandra |
| Sized by | Total history — terabytes | Entity count × feature size — far smaller |
The split is forced by the access patterns, and the online store is much smaller than people expect
Lesson 5 established that columnar formats optimize scanning many rows of few columns, and key-value stores optimize fetching one row of many columns. A feature store needs both, so it has both. Forced, not chosen.
The sizing consequence is worth working out, because it makes the online store far cheaper than it sounds:
OFFLINE: every feature value, every entity, for the full training history
-> grows with TIME. Terabytes, and growing daily.
ONLINE: the latest value only, per entity
-> does NOT grow with time. Bounded by entity count.
10M users x 200 features x ~20 bytes = ~40 GB
-> fits comfortably in memory
The online store is bounded by cardinality, not by history, which is exactly why an in-memory key-value store is affordable for it and would be absurd for the offline store.
This is the same insight as the typeahead chapter's 60 GB trie: when the working set is bounded by the number of entities rather than by the volume of events, memory-resident storage becomes the obvious choice.
The single pipeline
'Same logic, two destinations' is the design's central claim, and it prevents only one of three skews
"A single feature computation pipeline, powered by tools like Feast, writes to both stores using the same logic. This prevents training-serving skew, ensuring features behave identically in both environments."
The mechanism is right and the claim is too strong. A single implementation eliminates implementation skew — the failure in the chapter's own quiz, where Spark and Flink compute avg_spend_30d differently.
It does not address the other two, and Lesson 8 covers both in full:
IMPLEMENTATION skew two code paths compute the same feature differently
-> FIXED by a single pipeline ✅
FRESHNESS skew identical code, different refresh cadence
-> offline recomputed in a 4-hour batch window
online updated continuously
-> the VALUES differ even though the LOGIC is identical
-> NOT fixed ❌
POINT-IN-TIME skew training joins a feature value computed AFTER the label
-> the model learns from information that did not exist
at prediction time
-> NOT fixed, and it is the WORST of the three ❌
The second is a direct consequence of this design's own architecture: the processing layer runs on a 4-hour SLA (Lesson 2) while the stream path updates continuously. Same logic, different clocks, different values.
The third is the one that destroys models, and the chapter gestures at it twice — "point-in-time accurate training data generation" and "a critical capability here is preventing data leakage" — without ever explaining the mechanism.
A single pipeline is necessary and not sufficient, and saying so is what separates knowing that feature stores exist from knowing what they do.
The feature registry
A registry is what turns feature reuse from coordination into a lookup
The registry holds definitions, versions, owners, and computation logic — and each of those four fields corresponds to a failure it prevents.
Definition — one canonical statement of what avg_spend_30d means, so the divergence from Lesson 1 (calendar days versus rolling hours, refunds included or not) cannot happen silently.
Version — a model trained against v1 must not be served v2. Feature definitions change; the model that was fitted to the old one still exists. Versioning features is the same requirement as versioning datasets, and for the same reason.
Owner — when a feature breaks, someone is accountable. Without this, shared features become orphaned, and orphaned features are the ones nobody dares change and nobody dares delete.
Computation logic — this is the one that makes it enforceable rather than documentary. If the registry holds the logic, consuming a feature means referencing it, not reimplementing it. A registry that holds only descriptions is a wiki.
A registry with the logic in it is a contract; a registry with only metadata is documentation — and Lesson 6 showed that documentation loses to three processing engines.
The relationship to Lesson 5's metadata catalog: the catalog indexes datasets, the registry indexes features. Same idea, different granularity, and the design needs both because a table is not a feature — a feature has a definition, a freshness, an owner, and a version, none of which a table schema captures.
The schemas
Offline storage
user_id feature_name feature_value event_timestamp 101 avg_txn_amount 250.75 2025-10-15T09:00Z
Online storage
key (user_id) feature_dict 101 {"avg_txn_amount": 250.75, "txn_count_7d": 12}
The offline schema is entity-attribute-value, which defeats the columnar format it sits in
The layout is EAV — one row per (entity, feature, timestamp) — and the design pairs it with a note that "columnar formats in the offline store are optimized for large-scale analytical queries (OLAP), the primary workload for training."
Those two things work against each other.
Columnar formats win by storing each column contiguously so a scan reads only the columns it needs and compresses well because adjacent values are similar. EAV destroys both properties:
EAV: feature_value is ONE column holding every feature's values
-> avg_txn_amount (250.75) next to txn_count_7d (12) next to
is_premium (true) — heterogeneous, so compression is poor
-> selecting 3 of 200 features still scans ALL rows and filters
-> every training query needs a PIVOT to produce a feature matrix
WIDE: one column per feature
-> select exactly the 3 columns needed
-> each column homogeneous -> excellent compression
-> the layout IS the training matrix, no pivot
There is a real reason people reach for EAV — adding a feature requires no schema change — and it matters when features are added constantly. But modern table formats support schema evolution (Lesson 5), which is precisely the capability that removes the motivation.
Two further problems with the row as written:
feature_value has no type. One column holding floats, integers, booleans, and embeddings means everything is a string or a variant, and type errors surface at training time.
There is no feature version column, despite versioning being a stated requirement and the registry tracking versions. The offline store cannot answer "give me avg_txn_amount as defined in v1."
Store features wide, one column per feature, and let schema evolution absorb new ones — the layout should be the training matrix.
The online schema stores a blob, which brings back the lost update
key (user_id) | feature_dict
101 | {"avg_txn_amount": 250.75, "txn_count_7d": 12}
Storing all of an entity's features as one serialized value creates two problems.
Every update is a read-modify-write of the whole blob, which is the lost update from the payment chapter, now applied to features:
Pipeline A computes avg_txn_amount: reads the dict, sets it, writes back Pipeline B computes txn_count_7d: reads the dict, sets it, writes back -> whichever writes second overwrites the other's field -> a feature silently reverts to a stale value
And a stale feature value is, by definition, training-serving skew — arriving through a concurrency bug rather than a logic difference.
Every read fetches and parses all features, even though getFeatures(entity_id, feature_names) takes a subset. At 15,000 RPS, deserializing 200 features to return 3 is wasted CPU on the latency-critical path.
The fix is the store's native structure rather than a serialized blob:
Redis HASH: HSET user:101 avg_txn_amount 250.75 <- field-level write
HMGET user:101 avg_txn_amount txn_count_7d <- field-level read
DynamoDB: one attribute per feature, UpdateExpression per attribute
Cassandra: one column per feature
Field-level writes eliminate the lost update; field-level reads eliminate the waste.
Never store independently-updated values in a single serialized blob — the same conclusion Lesson 5's mutable-balance discussion reached, and the same fix.
The APIs
listFeatures() -> discovery
getFeatures(entity_id, feature_names) -> ONE entity, latest
getBatchFeatures(entity_ids, feature_names,
start_time, end_time) -> history, for training
serveFeatures(model_id, entity_ids) -> MANY entities, latest
The API models many entities per request and the estimate models one
Notice the asymmetry: getFeatures takes a single entity_id; serveFeatures takes an array.
The array is the realistic one, and it changes the sizing. Real inference is rarely one entity:
Recommendation: score 500 candidate items for one user Search ranking: score 200 documents Fraud: one transaction, but features for user + merchant + device
Lesson 2's estimate was 15,000 RPS × 2 KB = 30 MB/s, which models one feature vector per request. With batch lookups:
15,000 requests/s x 500 entities = 7.5M feature lookups/second
That is a completely different sizing problem for the online store — and it is why real feature stores emphasize multi-get as the primary operation and why the online store must be memory-resident.
When the serving API accepts an array, the request rate is not the lookup rate, and the estimate should be built from the second.
The bright side: multi-get is cheap in exactly the stores named. A Redis pipeline or a DynamoDB BatchGetItem amortizes the round trip, so 500 lookups is not 500 round trips — which is the reason the design is viable despite the estimate not showing it.
getBatchFeatures is the point-in-time API, and its signature hints at it
getBatchFeatures(entity_ids, feature_names, start_time, end_time)
The time window is what makes this the training API rather than a bulk version of getFeatures. Training data is not "the current value for these users" — it is "the value each feature had at the moment of each training example."
The description says as much: "returns a dataframe of values at specific past timestamps."
That is the mechanism Lesson 8 covers in full, and it is why the offline schema carries event_timestamp. Without it there is no way to reconstruct history, and training silently uses present-day values for past events — the leakage the chapter names and never explains.
Changing a feature definition means rewriting history
Feature definitions change — a bug is fixed, a window widens, a new signal is added. The awkward part is that a model trains on history, so a changed definition invalidates every past value.
Backfill is recomputing a feature over history so the training set is internally consistent. Without it you train on a mix of two definitions, which is a subtle and very hard-to-diagnose data bug.
Version rather than overwrite. Overwriting makes past training runs unreproducible — a model that scored well can never be rebuilt, and you lose the ability to attribute a regression to a data change rather than a code change. Writing a new version and letting models pin one keeps history intact.
Backfills are also expensive and easy to underestimate: recomputing one feature across years of events is a full pass over the raw zone, which is exactly why raw data is retained immutably in the first place.
Key takeaway
The dual store is forced by opposite access patterns, and the online half is far cheaper than it sounds because it is bounded by entity cardinality rather than by history — the same insight as the typeahead trie. The central claim, one pipeline writing both stores, eliminates implementation skew and not the other two kinds: freshness skew follows directly from this design's own 4-hour batch versus continuous stream, and point-in-time skew is worse still. A registry that holds the computation logic is a contract; one that holds only metadata is documentation — and documentation loses to three processing engines. Two schema choices undermine the layer: EAV defeats the columnar format it sits in (store features wide, one column each), and a serialized feature blob reintroduces the lost update, where a stale field is skew. Finally, when the serving API takes an array, the request rate is not the lookup rate.
Next: training-serving skew in full.