The Problem It Solves
In one line: every feature in a production model is implemented twice, and a feature store exists to make that once.
The two code paths
A model needs "average order value over the last 30 days". That number gets computed in two completely different places.
For training, a batch job reads the warehouse, aggregates over historical orders, and produces one row per training example. Written in SQL or Spark, by a data scientist, optimised for throughput over millions of rows.
For serving, a service computes the same quantity for one user, right now, inside a latency budget. Written in application code, by a backend engineer, optimised for a single-digit-millisecond lookup.
Same name, two implementations, different languages, different teams, different review processes. They agree by convention and drift by default.
How they drift
The failures are mundane, which is what makes them survive code review.
| Divergence | Example |
|---|---|
| Window boundary | Training uses 30 calendar days; serving uses rolling 720 hours |
| Timezone | Batch runs in UTC; the service uses the user's local day |
| Inclusion rule | Training counts cancelled orders; serving excludes them |
| Null handling | Batch drops nulls before averaging; serving imputes zero |
| Rounding | One rounds to cents, the other keeps floats |
| Late data | The warehouse has orders that arrived after the fact; the live path never sees them |
Each is a small, defensible choice. Together they produce a model trained on one distribution and served another — and nothing errors. Offline metrics are computed on the training path, so they look fine. Production quality is worse and the cause is invisible.
That gap is training-serving skew, and a feature store's central claim is that it is better prevented by architecture than by discipline.
Why discipline does not work
The obvious objection: just write the transformation once in a shared library and call it from both.
That helps and it is often the right first step. It does not close the gap, because the two paths differ in more than the transformation:
The inputs differ. Batch reads a warehouse table that has been compacted, deduplicated and corrected. Serving reads a live operational store. Same logic over different inputs gives different answers.
The execution differs. A Spark aggregation over a partition and a Redis lookup are not the same computation even when they claim the same semantics.
Time differs. This is the deep one. Training computes a feature over historical data, which means it must reconstruct what the value was at some past moment. Serving computes it over current data, which is simply what is true now. Those are different problems, and the historical one is where the correctness bugs live.
What a feature store actually provides
Five things, and it is worth being able to name them rather than saying "it stores features".
A registry — the definition, singular. One place a feature is described, with an owner, a version, and the entity it keys on.
Two stores — an offline store keeping history for training, and an online store keeping the latest value for serving.
Pipelines that materialise both from the same definition — so the two stores cannot disagree by construction rather than by agreement.
Point-in-time retrieval — the mechanism that reconstructs what a feature was at a past moment, which is the hard part.
Discovery and reuse — a feature computed by one team is visible to another, which is an organisational benefit rather than a technical one and is often the largest.
It is not a database
Worth saying because the name misleads. The stores are ordinary — a warehouse table and a key-value store. Neither is novel and neither is the point.
The value is in the coupling: that both are produced from one definition, that the offline one supports asking "what was this at that moment", and that a model can record which version of a feature it was trained on.
A team that adopts a feature store and keeps computing serving features in application code has bought a database and not solved the problem.
Key takeaway
Every production feature is implemented twice — a batch job for training and application code for serving — in different languages, by different people, over different inputs. They agree by convention and drift by default, through window boundaries, timezones, null handling and late data, and none of it errors. A feature store's claim is that a single definition materialised into both an offline history and an online latest-value store makes disagreement structurally impossible rather than merely discouraged. The stores themselves are ordinary; the coupling and the point-in-time retrieval are the product.
Next: whether you actually need one.