Evaluation

In one line: these compliance tables are better than most in the module — they name mechanisms rather than qualities. The gap is that one requirement conflicts with a foundational design decision, and neither table notices.

Functional requirements

These rows name components rather than qualities, which is the test most compliance tables fail

Worth crediting explicitly, because this module has seen the alternative repeatedly.

Compare with the previous chapters' tables — "redundancy is ensured with duplicated servers", "implementing disaster recovery protocols", "adhere to industry security standards". Those describe intentions. These rows name Kafka, columnar formats, metadata catalogs, and the feature store — things that either exist in the design or do not, and can be checked against the architecture.

The two rows carrying warnings are warnings precisely because the table is specific enough to check:

"Versioned features" is claimed and Lesson 7 showed the offline schema has user_id | feature_name | feature_value | event_timestamp — no version column. The registry tracks versions; the store cannot answer "as defined in v1."

"Monitoring" is answered with validation tools whose named examples are schema checks and uniqueness constraints — structural properties. The failures this platform is prone to are statistical: a null rate that tripled, a distribution that moved, a feature that stopped updating. Lesson 6 covers the gap.

A compliance row is only useful if it can be falsified against the design, and these can be — which is what makes the two failures visible.

Non-functional requirements

The reliability row names three mechanisms and each is the right one

"Acknowledgment mechanisms in ingestion, atomic commit protocols for data lake transactions, and automatic retries with replication."

Each maps to a real failure:

Acknowledgments in ingestion — the at-least-once guarantee from Lesson 4. A consumer that crashes mid-processing does not lose events. (The caveat from that lesson stands: at-least-once needs deduplication, or duplicates bias the training distribution.)

Atomic commit protocols for lake transactions — the table-format manifest swap from Lesson 5. This is the specific one, and it is the one most designs omit. Without it a failed Spark job leaves a partially written dataset that readers cannot distinguish from a complete one, and a model trained on a half-written table is trained on a silently truncated distribution.

Retries with replication — orchestrator-level retries plus object storage durability.

Naming atomic commits specifically is the tell that the author understood object storage's actual failure mode. It is a stronger row than most in this module.

GDPR erasure and an immutable raw zone cannot both hold, and the chapter asserts both

This is the evaluation's real gap, and it spans two sections that never meet.

NON-FUNCTIONAL REQUIREMENT:  "Enforce regulations like GDPR or HIPAA"
                             -> GDPR Article 17: the right to erasure

STORAGE DESIGN:              "Stores IMMUTABLE, raw data in its original
                             format for compliance and reprocessing"
                             -> preserved indefinitely, never modified

A user requests deletion. Their events are in the raw zone (immutable), the processed zone, the warehouse, the offline feature store, the online store, and every historical version retained for time travel and reproducibility.

And the conflict is not merely operational — it is between two requirements the chapter itself lists: reproducibility demands that data never change; erasure demands that it be removed.

The standard resolution is crypto-shredding, and it is worth knowing:

1. Encrypt each user's personal data with a PER-USER key
2. Store keys in a key-management service, keyed by user
3. On an erasure request, DELETE THE KEY

  -> the ciphertext remains in every immutable zone, unchanged
  -> it is permanently unreadable
  -> file layouts, partitions, and historical versions are untouched
  -> the data is erased in the sense the regulation means

Two supporting practices make it workable: separate personal data from behavioural data, so features derived from anonymized or aggregated events survive erasure and remain useful for training; and maintain a deletion log, since re-ingesting from an upstream source that has not yet processed the deletion would resurrect the record.

When immutability and erasure are both required, make the data unreadable rather than absent. The chapter names both requirements and never reconciles them — and this is the answer an interviewer is listening for.

Scalability is genuinely easy here, and it is worth knowing why

The row is short and correct, and the reason it can be short is structural: almost nothing in this platform holds state that constrains scaling.

The one real constraint is shuffle (Lesson 6): adding nodes adds compute and adds shuffle partners, so processing throughput is sublinear in node count and the network eventually binds.

Compare with the module's other systems, where scaling meant choosing a partition key and living with its skew — user IDs and celebrities, prefix ranges and letter frequency, hosts and domain concentration. Here the partition key is time, and time distributes perfectly by construction.

A platform whose work is batch, whose partition key is time, and whose state is object storage scales almost trivially — which is why the interesting problems in this chapter are all about correctness rather than capacity.

What the evaluation omits

Five things a strong answer adds, each covered earlier:

The processing SLA. The 4-hour window is listed in the assumptions and never applied — a 6× understatement, offset by a numerator that counts outputs as inputs (Lesson 2).

The internal network flow. Shuffle traffic dwarfs the ingestion and serving bandwidth the estimation does compute, and it is what sizes the network (Lessons 2, 6).

Two of the three skews. Freshness skew follows from this design's own 4-hour batch versus continuous stream; point-in-time skew needs an as-of join. The single-pipeline claim addresses neither (Lesson 8).

The CDN. Placed in front of feature data, where a stale value is the problem the platform exists to prevent (Lesson 9).

Drift. Listed as a challenge in Lesson 1 and answered nowhere — and it needs the statistical checks the monitoring row does not name (Lessons 6, 8).

Next: the interview walkthrough.