Storage: Zones, Formats, and the Catalog

In one line: the zone model is the most copied pattern in data engineering, and the reason for it is a single idea — you cannot know today every question you will ask of this data tomorrow.

The zones

Schema-on-read is a bet about the future, and it is the right one for ML

The two disciplines sound like a preference and are really a decision about when you commit.

SCHEMA-ON-WRITE: define the schema first, reject anything that doesn't fit
  -> data is always clean and queryable
  -> anything not anticipated is DISCARDED AT THE DOOR

SCHEMA-ON-READ:  store everything as it arrived, impose structure at query time
  -> data may be messy
  -> nothing is lost

For a warehouse serving known reports, schema-on-write is right — you know the questions.

For ML you do not. The feature that matters most next year is in a field nobody thought to keep this year. A model built to predict churn suddenly needs a device-type field, or a referrer, or a millisecond timestamp that was rounded away at ingestion because it "wasn't needed."

So the raw zone keeps everything, and the cost is that raw data is not directly usable. Which is fine, because it is not meant to be queried — it is meant to be reprocessed.

Schema-on-read is insurance against not knowing your future questions, and in ML you structurally cannot know them, because feature engineering is the process of discovering which fields matter.

The chapter's second reason — compliance — is the weaker of the two and worth flagging: keeping every raw record forever is exactly what collides with GDPR erasure. Lesson 10 covers the resolution.

The raw zone being immutable is what makes reprocessing possible at all

Stores immutable, raw data in its original format for compliance and reprocessing.

Immutability here does more work than the word suggests, and it directly serves the reproducibility challenge from Lesson 1.

Reprocessing is a routine operation in ML, not an exception. A feature definition changes, a bug is found in a transformation, a new model needs a longer history. Each requires re-deriving processed data from raw:

raw (unchanged, forever)
  -> reprocess with v2 logic -> processed_v2
  -> reprocess with v3 logic -> processed_v3

If the raw zone were mutable, you could not do this — reprocessing would produce a different answer each time, and you could never tell whether a change in results came from your new logic or from the input having shifted.

This is the same reasoning as the immutable ledger in the payment chapter and content-addressed artifacts in the deployment chapter: an append-only source of truth means every derivative can be recomputed and verified, and corrections are new derivations rather than edits.

Make the design immutable and everything downstream becomes a reproducible function of it. That single property is what turns a data lake from a dumping ground into infrastructure.

Formats and table formats

Cleaned datasets move to processed zones, which use modern table formats to support ACID transactions, time travel, and schema evolution.

Three properties, and each solves a specific failure of plain files on object storage

"Modern table formats" means Delta Lake, Apache Iceberg, or Hudi — a metadata layer over Parquet files. Each of the three properties fixes something that plain files cannot do.

ACID transactions. Object storage has no notion of a multi-file transaction. Without one, a Spark job writing 500 Parquet files fails halfway and readers see a partially written dataset — some new files, some old, no way to tell. A table format maintains a manifest that is swapped atomically, so readers see the old version or the new one, never a mixture.

That is the atomic pointer swap pattern again — that building block's trie publication, that building block's blue-green deployment. Build the new version beside the old and flip one reference.

Time travel. Query the table as it was at a timestamp or version. This is not a convenience feature here — it is how reproducibility is implemented. A training run pins a table version, and the same query returns the same data a year later.

Schema evolution. Add a column without rewriting terabytes of history. Essential when feature engineering is iterative by nature.

A table format is what makes a directory of files behave like a database, and all three properties trace back to the same trick: an atomically-updated manifest describing which files constitute the current version.

Columnar is right for one workload and wrong for the other, and the design uses it for both

Parquet is columnar — values from one column stored contiguously — and the payoff is large for analytics:

Query: SELECT AVG(amount) FROM transactions WHERE year=2025
ROW-oriented:     read every column of every row, discard 95%
COLUMNAR:         read ONLY the amount column
                  + compress well (like values adjacent)
                  + vectorized execution

Training reads a few features over millions of rows, so columnar is exactly right for the offline path.

But note where it is wrong: a single-entity lookup. Fetching all features for user 101 in a columnar store means touching every column file to reassemble one row — the worst case for the format.

Which is precisely why the online feature store is not columnar but a key-value store. Columnar formats optimize scanning many rows of few columns; key-value stores optimize fetching one row of many columns, and a feature store needs both, which is the whole argument for the dual-store split in Lesson 7.

Partitioning by directory

Deeper levels use hive-style partitioning (year=.../month=...). This transforms the directory structure into an indexed column, enabling query engines to skip irrelevant folders (partition pruning).

s3://lake/
  raw/
    events/
      year=2025/month=10/day=15/hour=09/
        part-0000.json
  processed/
    transactions/
      year=2025/month=10/day=15/
        part-0000.parquet

Encoding a column into the path is a free index, and the trade-off has a name

The mechanism is simple and the effect is large: because the partition value is in the path, a query engine reading WHERE year=2025 AND month=10 can eliminate every other directory without opening a single file.

Without pruning: list and read 365 days of files, filter in memory
With pruning:    read ONE directory

That is an index that costs nothing to maintain, because the filesystem layout is the index.

The trade-off is the standard one and it is worth being able to name: the small files problem.

TOO COARSE (year=):   pruning barely helps — you still scan a year
TOO FINE  (hour=, and partitioned by user too):
  -> millions of tiny files
  -> metadata operations dominate; each file has fixed read overhead
  -> Spark spends more time listing files than reading data

The rule of thumb is to target partitions in the hundreds of megabytes to low gigabytes, and to partition on the column your queries actually filter by — which for this platform is time, since both training windows and batch jobs are time-scoped.

Partition by the dimension your queries filter on, and size partitions so that metadata overhead stays negligible. Choosing a high-cardinality key like user_id is the classic mistake — it produces perfect pruning and unusable file counts.

The metadata catalog

A central metadata catalog manages metadata across this multi-layered architecture. It stores schemas, table definitions, and version histories across both the data lake and the warehouse, providing a unified view of datasets.

The catalog is what makes a lake discoverable rather than a swamp

Without it, a data lake is a directory tree that only its authors can navigate — the failure mode with a name, "data swamp."

The catalog answers the questions that otherwise require asking a person:

What datasets exist?          -> discovery
What is in this one?          -> schema
Who owns it?                  -> accountability
Where did it come from?       -> lineage
How big / how fresh is it?    -> statistics

Two things make it structural rather than documentation.

It spans the lake and the warehouse. Data lives in both, and without a unified catalog each has its own view and neither knows about the other's.

Statistics feed query planning. Row counts, min/max per partition, and null counts let engines skip files before reading them — so the catalog is a performance component, not just an inventory.

And note the relationship to Lesson 1's challenges: the catalog answers discoverability, while the feature registry in Lesson 7 answers ML-specific reusability. They are the same idea at different granularity — a catalog indexes datasets; a registry indexes features — and the design needs both because a table is not a feature.

Why a warehouse in addition to a lake

The design keeps both a processed data lake and a data warehouse, which looks redundant — both store cleaned, columnar, queryable data.

The honest justification is users, not capability. Modern engines query Parquet on object storage almost as well as a warehouse does. What a warehouse adds is managed performance, mature SQL, BI-tool integration, and fine-grained access control — which matter to analysts and matter much less to a Spark job.

So the split is: data engineers and ML pipelines work against the lake; analysts and dashboards work against the warehouse. The chapter's own processing layer reflects this exactly, running Spark for the lake and dbt for the warehouse.

That is a defensible reason and it is worth stating as such, because "we have both a lake and a warehouse" without a reason is how platforms accumulate duplicate copies of the same data with divergent definitions — the very inconsistency the feature store exists to prevent one layer down.

Next: the processing layer.