The Processing Layer

In one line: two transformation engines look like duplication and are not, and the quality-gate idea here is the one that most directly protects the models downstream.

The layer

Spark, and why it fits

Spark processes these workloads by partitioning massive datasets across a cluster, enabling parallel in-memory execution. It automatically manages the complexity of data shuffling for aggregations and joins, while its DAG scheduler optimizes execution plans and ensures fault tolerance by recomputing lost partitions if a node fails.

Recomputing lost partitions is a different fault-tolerance model from anything else in this module

Every other system in this module handles failure by replicating state: replicate the queue, replicate the database, keep a standby environment. Spark does something else.

It keeps the recipe rather than the result. Each partition of data is described by the sequence of transformations that produced it — the lineage in the DAG. When a node dies:

REPLICATION model:  the data existed in 3 places -> read another copy
LINEAGE model:      the data existed in 1 place -> RE-DERIVE it from its inputs

That works because the transformations are deterministic functions of immutable inputs — which is exactly the property Lesson 5's immutable raw zone provides. Fault tolerance by recomputation is only available when your inputs cannot change underneath you.

The trade is memory and time for storage: no replicas to keep, but a failure costs a recomputation, and a long chain of transformations can cost a lot of recomputation — which is why real pipelines checkpoint periodically, truncating the lineage.

When inputs are immutable and transformations are deterministic, lineage is cheaper than replication. That is a genuinely different answer to the fault-tolerance question than any other chapter gave.

Shuffling is named in passing and it is the dominant cost

"It automatically manages the complexity of data shuffling for aggregations and joins."

"Automatically manages" is doing a lot of work in that sentence. A shuffle is what happens when data must be redistributed across the cluster so that all rows with the same key land on the same worker — which every GROUP BY and every non-broadcast join requires:

Before: rows for user 101 scattered across 15 nodes
After:  ALL rows for user 101 on ONE node
  -> every node sends data to every other node
  -> all-to-all network transfer, plus disk spill when it exceeds memory

This is the traffic Lesson 2 identified as the largest flow in the system and never quantified — shuffle volume commonly exceeds the input volume, so a job reading 1.5 TB can move several terabytes across the network internally.

Two consequences that matter for the estimate:

Shuffle is why processing capacity is not linear in node count. Adding nodes adds compute and adds shuffle partners; past a point, network becomes the bottleneck.

Shuffle is where skew hurts most. Lesson 2's 20% buffer was justified by data skew, and skew manifests during shuffle: one key with disproportionate rows means one node receives disproportionate data, and the job waits for it.

In distributed processing the expensive operation is moving data, not transforming it — which is why avoiding shuffles (broadcast joins, pre-partitioned data, partial aggregation) is the main lever on batch performance.

Two engines: Spark and dbt

In parallel, SQL-centric transformations that occur directly within the data warehouse are managed using dbt, which provides modular modeling, version-controlled transformations, and lineage-aware execution.

ETL and ELT are two orders of the same letters, and the ordering is the decision

The two engines correspond to two patterns, and the difference is where the transformation happens.

ETL (Spark):  Extract -> TRANSFORM (in the cluster) -> Load
              transformation happens OUTSIDE the destination store

ELT (dbt):    Extract -> Load (raw into the warehouse) -> TRANSFORM (in SQL)
              transformation happens INSIDE the destination store

Neither is better; they suit different work:

So the split is by who writes the transformation and what it operates on, which mirrors the lake/warehouse split from Lesson 5 exactly: data engineers and ML pipelines on the lake with Spark; analysts on the warehouse with dbt.

Two transformation engines is justified when there are two populations of authors with different languages and different targets — and it is unjustified when it is just two teams that never agreed. The tell is whether the same logical transformation exists in both, which is the failure the next callout describes.

Two engines over the same data is how feature definitions diverge — and the chapter's own quiz says so

The design runs Spark, dbt, and (from Lesson 4) Flink for stream processing. Three engines, all capable of computing aggregations over the same underlying data.

That is exactly the setup for the failure the chapter poses as its closing quiz:

"avg_spend_30d is being calculated by two different services: the main Spark (ETL) job and a separate, older Flink (streaming) job. This causes two different values for the same feature. What fundamental component was bypassed?" → The feature store.

The answer is right, and the tension is worth naming: the architecture that makes this failure possible is the architecture the chapter recommends. Multiple engines are justified — but each additional engine is another place a feature definition can be implemented.

The discipline that resolves it is the "dual-publisher" rule from Lesson 3, stated as a constraint rather than a description:

Features are computed by ONE pipeline, registered in the feature registry,
and consumed by REFERENCE — never reimplemented in another engine.

Spark / dbt / Flink may compute DATASETS.
Only the feature pipeline computes FEATURES.

Multiple processing engines are fine for datasets and fatal for features, because a dataset that differs between engines is a discrepancy someone notices, and a feature that differs is a model that quietly degrades.

Data quality as a pipeline stage

The infrastructure embeds Great Expectations within pipelines. Every dataset is validated against explicit rules (schema checks, uniqueness constraints) before being propagated downstream. Intermediate outputs and quality metrics are stored for auditability.

'Before being propagated downstream' is what makes this a gate rather than a dashboard

The placement is the whole idea. Most data-quality tooling observes — it computes metrics, draws charts, and alerts. This design makes validation a stage that can fail the pipeline.

MONITORING:  bad data flows through -> a chart shows it -> someone notices ->
             by then it is in the feature store and in a trained model

GATING:      bad data fails the stage -> the pipeline stops ->
             downstream keeps the LAST GOOD version

The difference matters enormously here because of Lesson 1's core observation: ML failures are silent. A column that is 90% null when it is usually 2% null produces no error anywhere. It produces a worse model, six weeks later, attributed to drift.

A gate converts a silent data failure into a loud pipeline failure, which is the only way to catch problems whose symptoms are misattributed.

And the "stored for auditability" clause is the second half. Quality metrics over time are what let you answer "was this dataset already degraded when we trained on it?" — which is the first question in any model post-mortem, and unanswerable without a record.

The trade to acknowledge: a gate that fails stops the pipeline, and a stopped pipeline means stale features. That is usually the right trade — stale features are recoverable; a model trained on corrupt data is not — but it means the thresholds must be tuned, or the platform halts on noise.

What the quality checks named here do not catch

The examples given are schema checks and uniqueness constraints — structural properties. Lesson 4 already established that structural validation belongs at ingestion, so the interesting checks at this layer are the statistical ones, and the design does not name them:

STRUCTURAL (cheap, per-row):     types, nulls-not-allowed, uniqueness, ranges
STATISTICAL (needs the dataset): null RATE vs history
                                 row count vs expected
                                 distribution shift in a numeric column
                                 cardinality change in a categorical column
                                 freshness — is the newest record recent enough?

The statistical ones are what catch the failures that matter for ML, because a model is sensitive to distributions rather than to individual values. Every row can be individually valid while the dataset as a whole has moved.

This is also the mechanism for data drift, which Lesson 1 listed as a challenge and which appears nowhere else in the design: comparing a feature's distribution now against its distribution during training is the same computation as a quality check, run continuously.

Validate the distribution, not just the rows — in ML the aggregate property is the one the model depends on.

Orchestration

Both Spark and dbt pipelines rely on the workflow orchestrator to manage job scheduling, dependencies, retries, and alerting, ensuring processing runs consistently and reproducibly across environments.

Dependencies are the reason an orchestrator exists — the rest is cron

Scheduling alone is cron. What Airflow adds is the dependency graph, and in a data platform that graph is the system:

raw ingestion complete
  -> Spark cleaning job
      -> feature computation
          -> offline store write   -> model retraining
          -> online store write
      -> dbt warehouse models      -> BI dashboards

Each arrow is a real constraint. Running feature computation before the cleaning job finishes produces features from partial data — which is not an error, it is a subtly wrong number, exactly the silent failure mode this domain is prone to.

Two properties that follow, both of which the design names:

Retries with dependency awareness. A failed step blocks its dependents rather than letting them run on stale inputs. Compare with the alternative — independent cron jobs that each run at a fixed time and hope the upstream finished.

Reproducibility across environments. The DAG is code, version-controlled, and the same graph runs in dev, staging, and production — which is what makes the environment duplication from Lesson 2 meaningful at all.

An orchestrator's value is that it encodes what must happen before what, and in a pipeline where every stage feeds a model, running out of order is worse than not running.

Two paths or one: lambda versus kappa

Once you have both batch and streaming, the architectural question is whether they are two systems or one.

Lambda runs a batch path for correctness and a streaming path for freshness, merging at read time. It works, and its defining cost is that the same business logic exists twice in two different engines — which is precisely the mechanism that produces training-serving skew, since the batch path computes training features and the streaming path computes serving features.

Kappa keeps only the stream, and treats reprocessing as replaying the log from an earlier offset. One implementation, so no drift by construction. It demands long retention and enough throughput to replay history in reasonable time.

The connection worth drawing: the skew problem in the next lesson is a lambda-architecture problem. If one definition produces both paths, the skew cannot arise from divergent implementations — which is why feature stores push so hard toward a single definition compiled to both, or toward kappa outright.

Next: the feature store.