Features, Leakage, and Point-in-Time Correctness
In one line: a feature available in your warehouse is not necessarily available at the moment you have to decide, and training on the difference produces an excellent offline model that fails silently in production.
The bug
In your warehouse, a row about a transaction sits next to the account's total lifetime chargebacks. Train on it and the model looks superb — chargebacks predict fraud almost perfectly, because a fraudulent transaction causes the chargeback.
Serve it and the feature is either missing or computed after the event you are trying to predict. The model learned to read the answer.
The requirement has a name — point-in-time correctness: every feature in a training row must be reconstructed as of the moment the decision was made, not as of now.
Leakage is not always obvious
The chargeback case is easy to spot once stated. These are the ones that get shipped.
| Leak | Why it hides |
|---|---|
| An aggregate computed over the full dataset | The mean or the encoding saw the test set |
| A field back-filled by a later process | It exists in every historical row and in no live one |
| An ID that correlates with the label | Row order, batch number, or a sequential key that encodes time |
| A join against a slowly-changing dimension | The dimension table holds current values, not the values as of then |
| A duplicate of the same event | The same user in train and test makes memorisation look like generalisation |
The last two account for a large share of real incidents. A join against a "current" customer table is the single easiest way to leak the future into the past, and it looks like completely ordinary SQL.
Training-serving skew
Even with no leakage, training and serving can compute the same feature differently. That is training-serving skew, and it produces exactly the same symptom: strong offline numbers, weak production behaviour, no error anywhere.
There are two kinds, they look identical from the dashboard, and conflating them sends you looking in the wrong place.
| Feature skew | Distribution skew | |
|---|---|---|
| What differs | The definition — same name, different computation | The population — same definition, different data |
| Example | Training averages over 7 calendar days; serving over rolling 168 hours | Training data is from last quarter; the traffic mix has moved |
| Nature | A bug | A permanent condition |
| Fix | Compute it once, in one place | Monitor and retrain — it will recur forever |
Feature skew is fixed once. Distribution skew is never fixed; it is managed, and it gets its own lesson later in this chapter.
Why this makes a feature store architectural
Getting point-in-time correctness right by hand — for every feature, in both a batch training job and a low-latency serving path — is where teams lose months.
A feature store exists to make it structural rather than disciplinary. Features are defined once, materialised into an offline store for training and an online store for serving, and read through one interface, so the two paths cannot drift.
That is the whole argument for the component, and it is worth being able to make in two sentences. Candidates often name a feature store as a box on the diagram without being able to say what problem it solves; the problem is that training and serving are two different code paths reading two different stores, and nothing otherwise forces them to agree.
What it costs
A feature store is not free, and an interviewer may push on whether you need one. Honest answer: it adds a service to operate, a definition language to learn, and a hard dependency in the serving path. For a single model with a handful of features computed in one place, it is overhead.
It earns its place when you have several models sharing features, or a team large enough that the training path and the serving path are written by different people. That is the condition to name.
Freshness is a separate axis
Point-in-time correctness says the training value must match what was knowable then. Freshness asks how recent the serving value is, and it is a distinct decision with its own cost curve.
| Freshness | Typical mechanism | Cost |
|---|---|---|
| Daily batch | Scheduled job writes the online store | Cheap; useless for anything reactive |
| Hourly or minutes | Micro-batch or streaming aggregation | Moderate; a stream processor to operate |
| Seconds | Streaming with a low-latency store | Expensive; and now correctness depends on the stream |
| Request time | Computed inline from the request | Costs latency budget directly |
The right answer varies by feature within one model, and saying so is a good signal. A user's lifetime purchase count can be a day stale without harm. Their current session activity cannot be stale at all. Treating freshness as one global setting is the mistake.
Key takeaway
Point-in-time correctness is an architectural requirement, not a data-engineering detail: every training feature must be reconstructed as of the decision moment, or you build a model that reads the answer and fails silently in production. Distinguish feature skew, which is a bug you fix once, from distribution skew, which is a condition you manage forever. A feature store earns its cost when several models or several teams share features — its job is to make the training and serving paths structurally unable to disagree.
Next: why an offline improvement so often fails to appear online.