Why ML Data Infrastructure Is Different
In one line: every other chapter in this module designs a system with users. This one designs a platform other systems are built on, and its correctness criterion is unusual: the data a model sees in production must match the data it was trained on.
A platform, not an application
Rather than a single application, it is a foundational system supporting diverse workloads, including large-scale ingestion, transformation, feature storage, and low-latency inference.
Platform problems are graded differently from product problems
It is worth noticing what changes when the deliverable is infrastructure rather than a feature.
The users are internal and diverse. Data engineers ingesting sources, data scientists building features, ML engineers deploying models, analysts querying warehouses. Four groups with different access patterns and different tolerance for latency — which is why this design ends up with five distinct storage systems rather than one.
There is no single request path to optimize. A newsfeed has a hot path. This platform has a batch training path measured in hours and an online inference path measured in milliseconds, and they touch the same logical data. Keeping those two consistent is the entire subject of the chapter.
Failure is indirect. When this platform breaks, no user sees an error — a model quietly gets worse. That is a harder failure to detect than an outage, and it is why data quality and monitoring are functional requirements here rather than operational afterthoughts.
A platform's correctness is measured by what it enables downstream, not by its own uptime, which is why the requirements read as reproducibility, lineage, and consistency rather than as latency and availability.
The four challenges
General-purpose data warehouses or data lakes are not designed for the ML data life cycle, leading to critical issues.
| Challenge | What goes wrong | What the design owes it |
|---|---|---|
| Training-serving skew | Discrepancies between training and live data degrade model performance | The feature store — one definition, two stores |
| Lack of feature reusability | Teams recreate features, causing waste and inconsistency | The feature registry — a catalog with owners and versions |
| Reproducibility issues | Mutable data makes experiments unrepeatable | Versioned datasets, immutable raw zone, time travel |
| Scalability bottlenecks | Heavy transformations outgrow traditional systems | Distributed compute — Spark, partitioning |
Plus, in the design's own list: high latency in feature computation, monitoring for data drift, and the collaboration gap between data and ML teams.
Training-serving skew is the problem the whole architecture exists to solve
This is the headline, and it deserves a precise definition because the chapter's own quiz tests it: discrepancies between the data or features used for training and those used for live predictions.
Not a difference in hardware, not a difference in timing — a difference in the values themselves.
Why it is uniquely damaging: a model is a function fitted to a particular data distribution. Feed it inputs drawn from a different distribution and it is not slightly wrong, it is operating outside the domain it was fitted on — and it will still produce confident-looking outputs.
TRAINING: avg_spend_30d computed by a Spark batch job,
over a 30-day window ending at midnight
SERVING: avg_spend_30d computed by a Flink streaming job,
over a rolling 30-day window ending now
Same NAME. Same INTENT. Different VALUES.
-> the model was fitted on one and is asked to predict from the other
And the failure mode is the worst kind: silent. Nothing errors. No alert fires. The model's accuracy degrades, and the degradation is attributed to drift, or to the model being stale, or to the market changing — anything but the data pipeline.
A system whose failures are silent and whose symptoms are misattributed needs structural prevention rather than monitoring, which is why the answer is an architectural component — the feature store — rather than a dashboard. Lesson 8 works through the three distinct kinds of skew and what each requires.
Feature reusability is an organizational problem with a technical solution
Without a centralized platform, teams recreate features, causing waste and inconsistency.
The waste is the visible cost and the inconsistency is the expensive one.
Two teams both need "average spend over 30 days." They write it independently, and the definitions diverge in ways nobody notices:
Team A: calendar days, excludes refunds, NULL if no transactions Team B: rolling 720 hours, includes refunds, 0 if no transactions
Both are called avg_spend_30d. Both are defensible. Models trained on one and served the other silently break — which makes this the same failure as training-serving skew, arriving through an organizational route rather than a technical one.
The fix is a registry: features have names, owners, versions, and a single computation definition, and consuming a feature means referencing that definition rather than reimplementing it.
A shared definition with a single implementation is what turns a coordination problem into a lookup. The chapter's own closing quiz is exactly this scenario — the same feature computed by both a Spark job and an older Flink job — and the answer is that the feature store was bypassed.
Reproducibility requires immutability, and that sets up a conflict the chapter never resolves
Mutable data complicates experiment reproduction. Strict data and code versioning is essential.
Correct, and the reasoning is worth stating: reproducing an experiment means re-running the same code over the same data. If the underlying tables have been updated since, you cannot — you get a different model and cannot tell whether the difference came from your change or from the data moving underneath you.
So the design makes the raw zone immutable and datasets versioned, and training pulls a specific version rather than "current."
Hold onto that, because the non-functional requirements also name GDPR and HIPAA — and GDPR's right to erasure requires actually deleting a user's data on request. An immutable raw zone preserved indefinitely for reproducibility and a legal obligation to delete are in direct conflict, and the chapter names both without noticing. Lesson 10 covers the standard resolution.
What the platform must do
| Functional requirement | Detail |
|---|---|
| Data collection | Ingest from databases, real-time event streams, APIs, and logs |
| Processing and transformation | Raw → model-ready: cleaning, normalization, feature engineering |
| Batch and real-time handling | Historical batch for training and real-time streams for inference |
| Data storage | Raw, processed, and feature data in appropriate storage |
| Serving data to models | Versioned features and datasets, for training and inference, with low latency |
| Data monitoring | Data quality, lineage, and model-data interactions |
Non-functional: reliability · security and privacy (encryption, access control, GDPR/HIPAA) · scalability · performance · low latency (features in milliseconds).
Two requirements are doing structural work
Most of the list is standard. Two are not, and they generate the architecture.
"Batch and real-time handling" is the requirement that forces the dual-store design. Training wants complete history, high throughput, columnar scans over months of data. Inference wants the latest value for one entity, in milliseconds. Those are not the same access pattern and cannot be served well by one store — which is why the feature store has an offline half and an online half. Lesson 7 covers it.
"Serving versioned features" is the requirement that ties reproducibility to serving. Not just "give me the feature" but "give me the feature as it was defined at version N" — because a model trained against avg_spend_30d:v1 must not be served v2 when someone changes the definition.
When a requirement names two access patterns with different latency budgets over the same data, expect the design to split the storage rather than compromise. That is the same instinct as the fast-path/slow-path splits throughout this module, applied to storage rather than to compute.
The building blocks, and one that is wrong
Blob storage (the data lake — cost-effective, durable, for raw and processed data) · key-value store (the online feature store) · load balancers · message queue (buffer real-time streams, decouple producers from consumers, mitigate spikes) · CDN.
Four of those five are exactly right, and the CDN entry is not:
"CDN: Caches feature data geographically closer to inference models to reduce latency."
Lesson 9 covers this in full. The short version: feature values are per-entity (so the cache hit rate is near zero), constantly changing (the online store exists to hold the latest values), and a stale feature is training-serving skew — the exact problem the chapter opened with. Caching them contradicts the design's own thesis.
Key takeaway
This is a platform rather than an application, so it is graded by what it enables downstream — which is why reproducibility, lineage, and consistency appear as requirements where latency and availability would appear elsewhere. Training-serving skew is the problem the architecture exists to solve: a difference in the values a model sees at training versus serving time, whose failure is silent and routinely misattributed to drift — and silent, misattributed failures need structural prevention rather than monitoring, which is why the answer is a component. Feature reusability is the same failure arriving organizationally, fixed by a registry that turns coordination into a lookup. And reproducibility requires immutability, which the chapter never reconciles with the GDPR erasure obligation it also names.
Next: the estimation, and an SLA that is stated and then ignored.