Backfills
In one line: a new feature is useless until it has history, and building that history means running today's logic over a world that no longer exists.
Why you need one
A feature added today has no past. A model trained on twelve months of labels needs twelve months of that feature's values, and the only way to get them is to compute what it would have been.
Three situations force it, and they are common:
A new feature. The obvious case, and the most frequent.
A changed definition. The window went from 90 days to 30. Every historical value is now wrong for training against the new definition.
A corrected bug. The transformation had an error. Every value it produced needs recomputing, and any model trained on them is suspect.
Why it is the hardest operation
A backfill is not "run the pipeline over old data". It is running today's logic against yesterday's world, and the world has moved.
Each of these produces values that are internally consistent and historically wrong, and none of them errors.
The fourth is the most philosophically awkward. If your warehouse has since corrected historical orders, a backfill computes what the feature should have been rather than what it would have been. The serving system at the time saw the uncorrected data. Training on the corrected version teaches the model a cleaner world than the one it will operate in.
There is no universally right answer. There is a right decision, made deliberately, and recorded.
Streaming features are the worst case
A batch feature can at least be recomputed from a source that still exists. A streaming feature often cannot.
The stream may have a retention window of days. The pipeline's internal state — the running aggregate — was never persisted. And replaying months of events through a stateful pipeline is expensive and slow, if the events are even still there.
Doing one safely
The mechanics that make a backfill survivable.
Version the output, do not overwrite. Write backfilled values as a new feature version rather than replacing the existing ones. Then a model can be retrained on the new version, compared against the old, and rolled back if it is worse.
Backfill incrementally, oldest first, and checkpoint. These jobs take hours to days and will be interrupted. A resumable job with a cursor — the same discipline as the replay coordinator — turns an interruption into a delay instead of a restart.
Validate against a known window. Backfill a period where you already have correct values from the live pipeline, and compare. If the backfill disagrees with what production actually computed, the logic is wrong and you have found out cheaply.
That last one is the highest-value check and the most often skipped. It converts "we think the backfill is right" into evidence.
Bound the cost. A backfill over twelve months of a wide feature can be the most expensive job the team runs. Estimate it before starting, and be willing to narrow the window — six months of history is often enough, and half the price.
The consistency question during a backfill
While a backfill runs, the offline store holds a mixture: backfilled values for old periods, live-pipeline values for recent ones.
If the two disagree — and the validation step exists precisely because they might — a model trained across the boundary learns a feature whose meaning changes partway through its training window. That is a subtle and genuinely damaging failure, because the model may learn to associate the discontinuity with time-correlated labels.
The clean answer is to not train across a partial backfill. Either wait for completion, or restrict training to a period covered entirely by one version. Which is another argument for versioning the output rather than overwriting: with versions, "trained entirely on v2" is a checkable property.
When to skip the backfill
Sometimes the right answer is not to have history.
Train on a shorter window. If the feature exists for three months, train on three months. Less data, and every row is honest.
Introduce the feature going forward. Serve it, log it, and start using it once enough has accumulated — the log-and-wait pattern, applied deliberately.
Accept the feature is not yet available. A feature that cannot be backfilled correctly and has no history is a feature you cannot use yet, and saying so is better than shipping a model trained on reconstructed values nobody trusts.
Key takeaway
A backfill runs today's logic against a world that has moved — changed source schemas, changed entity definitions, reorganised reference data, and sources that have since been corrected — and every one of those produces values that are internally consistent and historically wrong without erroring. Streaming features are the worst case, because the events and the pipeline state may simply be gone, which is why a feature you expect to matter should be computed and snapshotted before anything uses it. Version the output rather than overwriting, checkpoint so an interruption is a delay, and validate against a window where the live pipeline already produced values — that check is what turns belief into evidence.
Next: making sure a model gets the feature it was trained on.