The Five Layers
In one line: four of these five layers are a conventional data platform. Recognizing which one is the ML-specific addition tells you where the design's real content is.
The architecture
| Layer | Component | Purpose |
|---|---|---|
| Ingestion | Message queue | Buffers high-throughput real-time streams (Kafka) to decouple producers from consumers |
| Stream / batch ingestion | Standardizes, validates, and batches incoming data before writing to storage | |
| Storage | Raw data lake | Immutable, original format (JSON, Avro) — for compliance and reprocessing |
| Processed data lake | Cleaned and optimized (Parquet), ready for analytics | |
| Data warehouse | Structured SQL engine (BigQuery) for high-performance analytics | |
| Metadata catalog | Schemas, lineage, and statistics for all datasets | |
| Processing | ETL pipelines | Clean, transform, aggregate |
| Data quality service | Validation checks to prevent bad data propagating downstream | |
| Workflow orchestrator | Schedule, dependencies, retries (Airflow) | |
| Feature store | Feature registry | Catalog of definitions, versions, owners, and logic — enables reusability |
| Offline storage | Historical values for point-in-time correct training datasets | |
| Online storage | Latest values only, low latency (Redis), for inference | |
| Serving | Serving API | Low-latency access to features and predictions |
Four of these five layers are a conventional data platform — only one is ML-specific
This is the observation that organizes the chapter.
Ingestion, storage, processing, and serving would appear, essentially unchanged, in any analytics platform built in the last decade. Kafka into object storage, Spark transforming into Parquet, a warehouse for SQL, an API on top. Nothing about that is about machine learning.
The feature store is the ML-specific layer, and it exists for one reason: the four challenges from Lesson 1. Look at the mapping:
| Challenge | The component that answers it |
|---|---|
| Training-serving skew | Offline + online stores fed by one pipeline |
| Feature reusability | Feature registry |
| Reproducibility | Point-in-time correct offline store |
| Scalability | Distributed processing — not ML-specific |
Three of the four challenges are answered inside the feature store, and the fourth is answered by Spark, which any platform has.
When a design adds one layer to a standard architecture, that layer is where the domain-specific problem lives — and everything else is supporting infrastructure it happens to need. Lessons 7 and 8 are about that layer; Lessons 4 to 6 cover the conventional four, which are still worth getting right.
The flow
- Ingestion — sources push through API connectors or publish to message queues. The ingestion layer is split into specialized stream and batch components.
- Raw storage — data lands in the raw lake with a schema-on-read approach, preserving original fidelity for compliance and reprocessing.
- Processing — ETL pipelines, managed by orchestration, execute cleaning and feature engineering. Data quality services validate; lineage is tracked.
- Processed storage — transformed data populates the processed stores and the warehouse. The metadata catalog is updated, making data discoverable.
- Feature store population — the processing layer extracts model-specific attributes. This layer acts as a dual-publisher.
- Serving — training pipelines query offline storage; live prediction queries online storage in milliseconds.
'Dual-publisher' is the single most important phrase in the design
Step 5 describes the feature store layer as "a dual-publisher", and Lesson 7's detail makes it explicit: "a single feature computation pipeline writes to both stores using the same logic."
That is the structural answer to training-serving skew, and the shape is worth naming:
ONE definition
-> ONE computation
-> TWO destinations (offline for training, online for inference)
Contrast with the failure mode it prevents, which is the chapter's own closing quiz — the same feature computed by a Spark batch job for training and an older Flink job for serving. Two implementations of one definition drift apart, silently.
Write once, publish twice, rather than computing twice. That is a pattern worth recognizing generally: whenever the same logical value must appear in two stores with different access characteristics, the risk is not the two stores — it is the two code paths.
The caveat, which Lesson 8 develops: a single pipeline eliminates implementation skew and not the other two kinds. Even identical logic produces different values if the two stores are refreshed on different schedules, and it produces wrong training data if the offline join is not point-in-time correct.
The two paths through the platform
Read the architecture as two paths with opposite requirements over the same data
The five-layer diagram makes this look like a pipeline. It is really a fork, and the two branches want opposite things:
| Training path | Inference path | |
|---|---|---|
| Latency | Hours are fine | Milliseconds |
| Volume per query | Months of history, millions of rows | One entity, a few KB |
| Access pattern | Scan — columnar, analytical | Point lookup — key-value |
| Consistency need | Point-in-time correct | Freshest available |
| Store | Parquet / BigQuery | Redis / DynamoDB / Cassandra |
No single store serves both well. A columnar analytical store answering single-key lookups in milliseconds is fighting its design; a key-value store scanning six months of history is fighting its.
So the dual-store architecture is forced, not chosen — the same conclusion the typeahead chapter reached about range partitioning and the collaborative editor reached about WebSockets. When two access patterns over the same data differ by orders of magnitude in latency and volume, you split the storage and accept the burden of keeping them consistent.
And that burden is precisely training-serving skew. The dual-store design creates the very problem the feature store exists to solve, which is why the "one pipeline, two destinations" rule is load-bearing rather than a nicety.
Where each layer's detail lives
| Lesson | Layer | The thing worth knowing |
|---|---|---|
| 4 | Ingestion | Two paths, and the schema registry as the compatibility gate |
| 5 | Storage | The zone model, schema-on-read versus schema-on-write, table formats |
| 6 | Processing | Spark versus dbt, orchestration, and quality as a pipeline stage |
| 7 | Feature store | The dual-store architecture and the registry |
| 8 | — | Training-serving skew in full — three kinds, and point-in-time correctness |
| 9 | Serving | Two access patterns, and the CDN mistake |
Key takeaway
Four of the five layers are a conventional data platform — Kafka into object storage, Spark into Parquet, a warehouse, an API — and only the feature store is ML-specific, which is where three of the four stated challenges are answered. When a design adds one layer to a standard architecture, that layer is where the domain problem lives. The architecture reads as a pipeline and is really a fork: a training path wanting hours, full history, and columnar scans, and an inference path wanting milliseconds and single-key lookups. The dual-store split is forced rather than chosen — and it creates training-serving skew, which is why the "one definition, one computation, two destinations" rule is the load-bearing idea in the whole design.
Next: the ingestion layer.