The Ingestion Layer

In one line: ingestion is the only layer that cannot refuse work, and it is where the design's most reusable idea sits — the schema registry as a contract between teams.

Two paths

Real-time events: high-velocity events (clicks, payments) are published to Apache Kafka. A stream processor (Flink or Spark Structured Streaming) consumes, validates, and enriches them before writing to object storage.

Batch data: large-volume data (database snapshots) arrives at scheduled intervals. Airflow manages extraction, cleaning, and formatting before loading into the raw zone.

The split is by data shape, and it determines everything downstream

Two paths look like duplication. They exist because the two data shapes have nothing in common operationally:

The asymmetry in the failure row is the important one. A batch job that fails can simply run again; a stream event that is dropped is gone, because the design has moved on and no one is holding a copy.

That is why the stream path — and only the stream path — has a message queue in front of it. Kafka is not there to be fast; it is there to be durable and replayable, so that a consumer failure does not equal data loss:

Producer -> Kafka (durable, retains for days) -> consumer
  consumer crashes -> restart from the last committed offset
  bug found later  -> REPLAY from an earlier offset and reprocess

That replay property matters more here than in any other chapter, because ML pipelines are frequently wrong and reprocessed — a feature definition changes, a bug is found, a model needs different history.

Buffer the path that cannot say "wait," and make the buffer replayable, because the transformation logic will change.

The schema registry

Both ingestion paths must ensure reliable delivery and schema compatibility. Tools like Confluent Schema Registry or validation libraries in Airflow prevent schema changes from breaking downstream systems.

This is a contract between teams, enforced by infrastructure

The schema registry is the most transferable idea in this layer, and its value is organizational as much as technical.

The failure it prevents is mundane and constant: a producing team renames a field, changes a type, or drops a column. Their service still works. Every downstream consumer breaks — and they break later, in a batch job at 3 a.m., far from the change that caused it.

A registry inverts that. Every message carries a schema ID; the registry holds the schemas and enforces a compatibility policy on registration:

BACKWARD compatible  new schema can read OLD data     -> consumers upgrade first
FORWARD compatible   old schema can read NEW data     -> producers upgrade first
FULL                 both                             -> either order works

So an incompatible change is rejected at registration time, by the producer's own deploy, rather than discovered by a consumer's failure.

Two consequences worth naming:

It moves the failure from consume-time to publish-time, which is the entire game — the error surfaces next to the change that caused it, and the person who can fix it is the person who sees it.

It is what makes schema-on-read safe. The raw zone stores data in its original form with no enforced schema, which sounds like it defers all validation. The registry means the shape was validated even though the storage is schema-less.

A schema registry converts an implicit cross-team contract into an enforced one, and it is the same instinct as the API contracts and proto files that govern service boundaries.

Validation happens twice, and the two checks are not the same check

It is easy to conflate the schema registry here with the Great Expectations data-quality checks in the processing layer. They catch different failures and both are needed.

SCHEMA validation (ingestion):   Is this the RIGHT SHAPE?
  -> field names, types, required fields, enum values
  -> a structural contract, checked per message, cheap

QUALITY validation (processing): Are these values SENSIBLE?
  -> nulls above a threshold, values out of range, uniqueness,
     row counts within expected bounds, distributions unchanged
  -> a semantic contract, checked per dataset, expensive

A field can be perfectly well-typed and completely wrong. age: 20000 passes any schema check. A column that is 90% null when it is usually 2% null passes every schema check ever written — and it will silently degrade every model that uses it.

Structural validation belongs at ingestion because it is cheap and per-message; semantic validation belongs at processing because it is statistical and needs the whole dataset. Lesson 6 covers the second.

Lineage metadata at the door

To support lineage and debugging, all ingested data is tagged with metadata (timestamps, Kafka offsets, batch IDs).

Tagging at ingestion is what makes the answer to 'where did this come from?' a lookup

This is a small practice with a large payoff, and it is worth doing for the reason the design implies rather than the one it states.

Lineage is usually described as a governance feature. Its practical value is debugging, and specifically debugging the failure mode from Lesson 1: a model got worse and nobody knows why.

Bad feature value spotted in production
  -> which rows produced it?
  -> which raw records did those come from?
  -> which Kafka offsets / which batch ID?
  -> which producer, at what time?

Without provenance stamped at ingestion, every step of that chain is an investigation. With it, each step is a query.

And the specific identifiers chosen are well judged:

Kafka offsets are exact and replayable — you can go back and read precisely the records in question.

Batch IDs do the same for the scheduled path.

Ingestion timestamps are distinct from event timestamps, and keeping both is essential. Lesson 8 shows why: the gap between when something happened and when you learned about it is the design of the hardest class of training-serving skew.

Attach provenance where data enters, because it cannot be reconstructed later — the same conclusion the payment chapter reached about dispute evidence and the deployment chapter reached about build provenance.

'Reliable delivery' is stated and the guarantee is never named

"Both ingestion paths must ensure reliable delivery and schema compatibility."

Reliable in which sense? The module has now established that this is a real choice with real consequences:

At-most-once   ack before processing  -> events LOST. Unacceptable here.
At-least-once  ack after processing   -> events DUPLICATED. The right default.
Exactly-once   not achievable end to end across independent systems

For an analytics and ML platform the answer is at-least-once, and then deduplicate — which needs something the ingestion layer is already collecting:

Every event carries a producer-assigned event_id
Downstream processing deduplicates on (event_id) within a window
  -> duplicates from retries and consumer restarts collapse

The consequence of not doing this is specific to ML and worse than it looks. Duplicated events do not just inflate counts — they bias the training distribution. A feature like txn_count_7d computed over a stream with duplicates is systematically high, and a model fitted on inflated counts will be served correct ones. That is training-serving skew arriving through the ingestion layer.

At-least-once delivery plus a non-idempotent aggregation equals a biased model, which is this domain's version of the conclusion the payment and deployment chapters both reached.

What happens to a record you cannot parse

Ingestion pipelines meet malformed data constantly — a schema change upstream, a truncated payload, a field that is suddenly null. The design decision is what the pipeline does when it cannot process one record.

Crashing is the naive failure: one malformed record from an upstream deploy stops every downstream consumer, and the outage lasts until someone is paged.

Silent dropping is the dangerous one, because it looks like success. Data disappears, the pipeline reports healthy, and the symptom surfaces weeks later as a model that quietly got worse — with no signal pointing at ingestion.

Dead-lettering takes the middle path: route the record and its error to a separate stream, keep processing, and alert on the rate rather than on individual failures. A trickle of bad records is normal; a sudden spike means an upstream contract changed, and that is the thing you actually want to know.

The rate is the signal, and it is worth saying so — a dead-letter queue nobody monitors is just a slower silent drop.

Next: the storage layer and the zone model.