Monitoring Feature Health

In one line: a feature pipeline that succeeds while producing nothing looks identical to one that is working, so job status is the wrong thing to alert on.

Four signals per feature

None of them needs labels, which is why they are the fast tier from the foundations module — they move within minutes, while model quality takes days or weeks to reveal itself.

Freshness is the one to build first. Not "did the job run" but how old is the newest value. A job that completes successfully having processed zero rows is the failure that hides behind a green pipeline dashboard, and value age is the only thing that catches it.

Null rate is the second. Nulls arriving from an upstream API outage or a renamed column are the most common data-quality failure, and a rate that jumps from 2% to 40% is unambiguous and instant.

Cardinality is the one people skip. A category feature that had 340 distinct values yesterday and has 12 today means the taxonomy changed or a join broke. Distribution monitoring on a numeric feature would catch its equivalent; for categoricals, cardinality is the equivalent.

The skew check

The monitor specific to this component, and the one that would have caught the failure the whole chapter is about.

This is the direct test. Log what serving used, recompute it through the training path, compare. A mismatch is skew, found by measurement rather than by a model getting worse three weeks later.

Two details make it workable:

Sample. Recomputing every served feature is as expensive as serving. A small random sample per hour is enough to detect a systematic divergence, which is the only kind that matters.

Allow for lag. The two stores legitimately disagree during the materialisation window, so the comparison has to be as-of a timestamp old enough that materialisation has completed. Comparing naively produces constant false alarms and the alert gets muted — which is worse than not having it.

What to alert on versus what to chart

Not every signal deserves a page.

Alert: freshness beyond its threshold, null rate crossing a bound, skew detected, a pipeline failing repeatedly. These are actionable and usually urgent.

Chart, and review: distribution drift, cardinality changes, usage counts, cost per feature. These are investigations rather than incidents — a distribution shift is often legitimate, and alerting on every one trains people to ignore the alerts.

That split matters because a feature store with hundreds of features generates a lot of signal. Alerting on all of it produces noise that gets muted, and a muted alert is worse than no alert because everyone believes it is covering them.

Per-feature thresholds

A global threshold does not work, and this is where the registry earns its place again.

A feature that is legitimately 90% null — an optional profile field — should not alert at 40%. One that is never null should alert at 1%. A daily feature is fine at 20 hours old; a streaming one is broken at 5 minutes.

So the expected ranges belong in the registry, next to the definition, set by the owner. That makes monitoring a property of the feature rather than a separate configuration someone forgets to update when the feature changes.

Cost and usage as health signals

Two non-obvious metrics worth carrying.

Usage. Which models read which features. A feature nobody reads is a pipeline running for nothing, and this is the number that makes deprecation possible.

Cost per feature. Compute plus storage plus serving throughput. Some features are enormously expensive for a small quality contribution, and without the number nobody ever finds out.

Putting cost next to usage occasionally produces the most valuable finding available here: a streaming feature costing real money, used by one model, contributing almost nothing. Nobody looks for that unless the numbers are on the same page.

Next: the walkthrough, where the whole thing gets designed live.