Resource Estimation

In one line: the processing estimate contains two errors that run in opposite directions and nearly cancel, which is exactly why the published number looks reasonable.

Storage

Daily raw ingestion: 1.5 TB/day (10,000 events/sec at 1.5 KB/event, plus overhead) Transformation factor: 2.5× raw (denormalization, enrichment, metadata) Feature data: 0.5× raw (aggregated and derived values)

10,000 events/s x 1.5 KB x 86,400  = 1.30 TB       -> 1.5 TB with overhead
Raw          1.5 TB
Transformed  1.5 x 2.5 = 3.75 TB
Features     1.5 x 0.5 = 0.75 TB
                        ---------
Total daily              6.0 TB
Annual       6 x 365   = 2.19 PB   (published 2.2 PB)

Every figure reproduces.

The 2.5x transformation factor is the interesting number, and it is the right shape

It is easy to read past, and it encodes something true about ML pipelines: transformation makes data bigger, not smaller.

The intuition from analytics is the opposite — aggregate, roll up, compress. ML feature engineering does the reverse:

Denormalization  join dimension tables INTO the fact table
                 -> a user ID becomes twenty user attributes
Enrichment       derive windows, ratios, encodings, embeddings
                 -> one transaction becomes fifty features
Metadata         timestamps, lineage tags, version markers on every row

Models want wide, flat, self-contained rows, because a training example must carry everything the model needs with no joins at read time. That is the opposite of a normalized schema, and it is why the processed zone is larger than the raw zone.

Note the countervailing factor the estimate does not mention: the processed zone uses columnar formats like Parquet, which typically compress 3–5× over raw JSON. So the 2.5× expansion and the columnar compression partly offset, and the honest figure depends on which dominates.

Feature data at 0.5× is correspondingly small, and correctly so — features are aggregates. It is the transformed zone that carries the bulk.

2.2 PB/year is the floor, and the design says so without following through

Note: Real-world systems require additional storage for versioning, redundancy, backups, and metadata.

Each of those is a multiplier, and they compound:

Base                          2.2 PB/year
x3 replication                6.6 PB
+ versioning / time travel    older snapshots retained for reproducibility
+ backups                     another copy, on another medium
+ dev/staging environments     (later revised to 2-3x)

Reproducibility is the expensive one and it is a stated requirement. Training on a versioned snapshot means keeping the snapshot — and modern table formats give you time travel precisely by retaining old file versions rather than overwriting.

The mitigation is the one every data lake uses: tier by age and by zone.

Raw zone, recent      hot object storage
Raw zone, older       infrequent-access / archive tiers
Processed zone        hot — this is what gets queried
Feature offline store hot for the training window, archived beyond it

When retention is driven by reproducibility rather than by usage, tiering rather than deletion is the lever — the same conclusion the payment chapter reached about regulatory retention.

Bandwidth

Average in  = 1.5 TB / 86,400              = 17.4 MB/s
Peak in     = 17.4 x 3                     = 52.2 MB/s
Serving out = 15,000 RPS x 2 KB            = 30 MB/s

All three reproduce. (The peak figure appears as 52.2, 52.5, and 53 in three places — immaterial, but worth noticing that the ingestion-server calculation uses the largest of the three.)

A 3x peak factor is a real design decision, and the design's own quiz explains it

"Why do we estimate based on peak throughput rather than daily average?" → "To ensure the system remains stable and does not drop data during natural traffic surges."

Correct, and worth being concrete about why it matters more here than elsewhere.

Ingestion is the one layer that cannot apply backpressure to its sources. A user's click happens whether or not your pipeline is ready. If the ingestion tier is sized for the average and traffic triples, you either drop events — permanently losing data that can never be reconstructed — or you have somewhere to put them.

That "somewhere" is the message queue, and it is why the block list includes one. The queue absorbs the spike so downstream sizing can be closer to the average:

WITHOUT a queue: every downstream stage must be sized for PEAK
WITH a queue:    the QUEUE absorbs peak; consumers drain at their own rate

So the 3× factor applies to the ingestion tier and the queue's retention capacity, and not to the processing tier — which is batch and inherently peak-insensitive.

Size the component that cannot say "wait" for the peak, and let it buffer for everything behind it. The same role the pub-sub layer played in the deployment chapter's build fleet.

The largest network flow in the system is the one that is never estimated

Note: The most significant network load is internal traffic between the storage layer and the processing cluster. This often reaches multi-gigabit-per-second speeds and requires a carefully designed network architecture.

The design names it and does not compute it, which is a pattern this module has seen repeatedly — that building block stated "2,000 machines per region" and never multiplied; that building block never computed the collaborator fan-out.

Here the figure is straightforward:

A Spark job reads 1.5 TB from object storage in a 4-hour window:
  1.5 TB / 14,400 s x 8 = ~830 Mb/s   just to READ

Plus SHUFFLE traffic between workers for joins and aggregations,
which for wide transformations commonly EXCEEDS the input volume,
and plus writing 4.5 TB of output back.

So the internal flow is an order of magnitude larger than the 52 MB/s ingestion and the 30 MB/s serving that the estimation does compute — and it is the one that dictates whether nodes need 10 GbE and whether the cluster should sit close to the storage.

In a data platform the external bandwidth figures are the small ones. The traffic that sizes your network is internal, and it is dominated by shuffle.

Servers

The ingestion per-node figure is unusually well justified

Per-node throughput: 10 MB/s. While a 1 Gbps NIC allows more, real-world throughput is limited by disk I/O, replication, and serialization.

This is the rare capacity assumption in the module that explains itself, and the explanation is correct. A 1 Gbps NIC is 125 MB/s; assuming a node ingests at line rate is the classic sizing error. What actually binds:

Disk I/O       Kafka persists every message; sequential writes, but still bounded
Replication    with RF=3, one incoming byte becomes ~3 bytes written and
               ~2 bytes of outbound network to followers
Serialization  encode/decode, compression, checksums — CPU per message

Replication is the one people forget, and it is a 3× amplification hiding inside the ingest path. The stated 10 MB/s already accounts for it, which is why the figure is an order of magnitude below the NIC.

Size an ingestion tier by its narrowest resource, not its network interface — and remember replication multiplies both writes and internal network traffic.

The processing estimate

Total daily data volume: 6 TB · Processing capacity per node: 500 GB/day · Processing SLA: data processed within a 4-hour window

6000 GB / 500 GB = 12 nodes, +20% buffer → 15 nodes

The SLA is stated in the assumptions and then ignored in the arithmetic — a 6x error

Read the three assumptions together. The capacity is per day. The SLA is four hours. The calculation divides them as if they were the same window.

Node capacity per DAY:      500 GB
Node capacity in 4 HOURS:   500 x 4/24  = 83.3 GB

Published:  6,000 / 500   = 12 nodes    <- assumes 24 hours
Correct:    6,000 / 83.3  = 72 nodes    <- honours the stated 4-hour SLA
                            ----------
Factor:                     6x

A 4-hour SLA is a 6× multiplier on the fleet, and the calculation silently spends 24 hours.

This is the third instance of the same shape in three chapters. that building block announced "we use the peak traffic load" and used daily active users. that building block announced "to estimate the server count, we use the peak traffic load" and substituted transactions-per-day-as-per-second. Here the assumption is listed in the same bullet list as the inputs and then not applied.

When a latency or window constraint appears in the assumptions, check that the denominator reflects it. A capacity expressed per day and a deadline expressed in hours must be reconciled before dividing.

And the numerator is the wrong quantity, in the opposite direction

"Total daily data volume: 6 TB (derived from storage estimates)"

The 6 TB came from the storage section, where it was:

6 TB = raw 1.5 (INPUT)  +  transformed 3.75 (OUTPUT)  +  features 0.75 (OUTPUT)

The transformed and feature data are what the processing job produces. They are not what it reads. A per-node figure of "500 GB/day of Spark/ETL throughput" conventionally describes input volume, so dividing the total stored volume by it counts the job's own outputs as additional work to be done.

The processing input is 1.5 TB/day. Now correct both errors together:

Published:               6,000 GB / 500 GB/day   = 12 nodes
Fix the WINDOW only:     6,000 GB / 83.3 GB/4h   = 72 nodes   (6x too high)
Fix the VOLUME only:     1,500 GB / 500 GB/day   =  3 nodes   (4x too low)
Fix BOTH:                1,500 GB / 83.3 GB/4h   = 18 nodes
                                                   +20% -> 22

The two errors run in opposite directions and very nearly cancel, which is precisely why 12 looks plausible and neither error was caught. The published 15 is not far from the corrected 22 — but it is right for entirely the wrong reasons, and the reasoning would not survive a change to either assumption.

Two compensating errors produce a defensible number and an indefensible model. Change the SLA to one hour, or the transformation factor to 5×, and the published method fails while the corrected one still works.

The 20% buffer is justified for the right reason

We add a 20% buffer for data skew, retries, and slowdowns to ensure SLA compliance.

Data skew is the correct headline concern for a Spark cluster, and it is worth knowing why. Distributed processing partitions by key; if one key holds far more rows than the rest, one worker does far more work and everyone waits for it:

Partition by user_id: most users have hundreds of events
                      a handful of bot accounts have millions
                      -> one task runs 100x longer than the median
                      -> the JOB's runtime is that task's runtime

Because a batch job finishes when its slowest task finishes, skew directly threatens a window-based SLA — which is exactly what the buffer is protecting.

Adding nodes helps only partially, though: more workers do not split a single hot partition. The real fixes are salting hot keys, adaptive partitioning, or broadcasting the small side of a skewed join. A capacity buffer mitigates skew's symptoms; only repartitioning fixes its cause.

The multipliers nobody totals

Growth buffer: 20–30% · Fault tolerance: N+1 or 1–2 extra per cluster · Environment duplication: 2–3× the server count for staging and development.

The stated multipliers compound to a fleet three to four times the base, and the chapter never computes it

Each factor is listed separately and they multiply:

Base clusters      6 + 15 + 30            =  51 nodes
+15% orchestration                        =  59
+30% growth buffer                        =  76
x2-3 environments                         = 152 - 229 nodes

The platform is three to four times the size of the sum of its parts, and that figure appears nowhere.

The environment multiplier is the one to push back on. 2–3× the full production fleet for staging and development is rarely right for a data platform, because the cost driver is data volume rather than request rate, and staging does not need production's data:

BETTER: staging runs the same CODE against SAMPLED data
        -> ~10% of the fleet, and pipelines still validate end to end
        -> plus ephemeral per-developer environments, created on demand

That preserves what staging is actually for — catching schema breaks, logic errors, and dependency failures — while paying a fraction of the cost. Duplicate the pipeline, not the petabytes.

When buffers are listed individually, multiply them out before quoting a number — the compounded total is the one that gets budgeted.

The estimation mistake specific to data platforms: sizing the raw feed and stopping. The same event exists in several zones at once, each is replicated, non-production environments duplicate the lot, and history is retained deliberately because models need to be retrained on the past.

Next: the five layers.