Versioning and Schema Evolution
In one line: a model is a function of a specific feature definition, so changing that definition changes the model — silently, and without redeploying it.
The failure
A feature's definition is improved. "Preferred category" moves from a 90-day window to 30 days, because 30 tracks recent interest better.
The pipeline is updated. The online store starts serving the new value. And every model trained on the old definition is now receiving a different quantity under the same name.
Quality degrades, no deployment happened, and the change that caused it was in a different repository owned by a different team. This is the hardest kind of regression to attribute, and it is entirely preventable.
Version the definition, not just the values
The rule: a change in what a feature means is a new version, not an edit.
| Change | Breaking? | Handling |
|---|---|---|
| Fix a bug producing wrong values | New version; retrain and compare before switching | |
| Change the window or aggregation | New version — the meaning changed | |
| Change the source table, same semantics | No, if values match | Validate on overlap, then swap in place |
| Add a new feature | Additive; existing models unaffected | |
| Widen a type | Usually not | Check downstream tolerance |
| Change null handling | Yes, and it looks harmless | New version — the distribution moved |
The last row deserves attention. Switching from dropping nulls to imputing zero looks like an implementation detail and changes the feature's distribution, which changes what the model learned. It reads as non-breaking and is not.
The general test: would a model trained on the old values behave differently on the new ones? If yes, it is breaking, regardless of how small the code change was.
Models declare what they need
The mechanism that makes versioning useful rather than merely recorded.
A deployed model carries a manifest of the features and versions it was trained on, and the serving path requests those specific versions. Then:
A new version does not affect existing models. They keep requesting what they trained on.
Migration is explicit. Retrain on the new version, compare, deploy the new model. Two deliberate steps.
Rollback works. Reverting a model reverts the features it uses, because the manifest is part of the model rather than of the environment.
Deprecation
Versioning creates a cost nobody plans for: old versions accumulate, and every one is a pipeline that runs and storage that fills.
The lifecycle worth stating:
Mark deprecated in the registry, with a reason and a replacement, so anyone discovering it sees the status.
Find the consumers — this is where usage counts earn their place. Deprecating a feature you cannot enumerate the users of is a guess.
Give a migration window, long enough for teams to retrain, which is weeks rather than days.
Stop computing, keep the data, then delete later. Halting the pipeline reclaims most of the cost immediately, and keeping the historical values allows a rollback.
A registry without deprecation becomes an accumulating list of features nobody dares delete, which is a mess that gets worse and never better.
Schema evolution in the sources
The version above the feature: your sources change too, and the store sits downstream of decisions it does not control.
A column is renamed. The pipeline breaks loudly. This is the good case.
A column's semantics change while the name stays. The pipeline continues, computing a different thing under the same name. Silent, and only distribution monitoring catches it.
A column becomes nullable. Values that were always present start being absent. If the transformation drops nulls, the population being aggregated silently shrinks.
The structural mitigation is a contract between the source and the store — an agreed schema with an owner, so a change is a negotiation rather than a surprise. That is organisational, and it is the only thing that reliably works. Failing that, distribution monitoring on the feature is the detector of last resort, which the next lesson builds.
Key takeaway
A model is a function of a specific feature definition, so changing that definition changes the model without anyone deploying it — the hardest regression to attribute, in a repository owned by someone else. Version definitions rather than editing them, and use the test "would a model trained on the old values behave differently on the new ones?", which catches null-handling changes that look harmless. Have models declare the feature versions they were trained on, so behaviour is closed over the model rather than over the environment and rollback works. Plan deprecation, since old versions accumulate as running pipelines. And agree schema contracts with sources, because a renamed column breaks loudly while changed semantics do not break at all.
Next: the serving path, and what to do when a feature is missing.