Chunking and Blob Metadata

In one line: chunk size is a genuine tuning decision with pressure from both directions, and the reasoning generalizes to almost every storage system you will design.

The three layers

Note: we generate unique IDs for user accounts, containers, and blobs using a unique ID generator.

Each layer is a list of the layer below — and that shape drives Lesson 9's partitioning

Read the Mapping column top to bottom: account → containers → blobs → chunks. Every level holds only the IDs of the level beneath it, which keeps each metadata record small and bounded.

It also explains a decision that is coming. If the three levels are sharded independently — account by account_ID, container by container_ID, blob by blob_ID — then listing one user's blobs means hopping across shards at every level.

Lesson 9 fixes exactly that by partitioning on the full path instead. Notice the tension now, because the interviewer will ask why the obvious sharding key is the wrong one.

Chunking

When a user uploads a blob, it is split into small chunks. This supports large files that exceed the capacity of a single contiguous disk block or data node. The manager node tracks chunk locations and assigns IDs to facilitate retrieval.

Metadata includes chunk IDs, assigned data nodes, and replica IDs. For a 128 MB blob split into two 64 MB chunks:

The system maintains three replicas for each chunk. During a write, the manager node uses free space metadata to select the primary and replica nodes. The replicas provide redundancy in case of node failure and can serve read requests to reduce load on the primary.

Count the columns — three replicas means four copies

This trips people up, and the table settles it: Datanode ID plus three replica IDs is four locations per chunk.

So "three replicas" means three replicas in addition to the primary. When Lesson 12 says the system maintains four replicas per blob, it is counting the same thing with the primary included — not a contradiction, a different convention.

Keep it distinct from the geographic count in Lesson 10 (a local copy, one in another data center, one in another region). That is a placement policy across three failure domains, a different axis from how many copies exist per chunk.

If asked, say it precisely: four copies of every chunk, placed across at least three failure domains.

Chunking solves three problems at once, only one of which is size

The stated reason is files exceeding a single disk block or data node, and that is real. But chunking pays off three more times in this design:

Parallel transfer. Lesson 7's client reads chunks directly and concurrently from different data nodes — four chunks means four simultaneous streams. Lesson 12 calls this out as the throughput mechanism.

Granular replication and repair. A failed data node needs only its chunks re-replicated, not whole blobs, and the sources are spread across the fleet rather than hammering one peer.

Even space utilization. Fixed-size units pack into free space cleanly. Variable-size whole blobs would fragment disks the way variable-size allocations fragment memory.

That is why chunking appears in GFS, HDFS, and every large object store — it is not primarily about big files, it is about making storage a fungible commodity.

The opposite problem: billions of tiny blobs

Everything above assumes blobs are large enough that splitting them is the question. Invert it — a photo service holding billions of 20 KB thumbnails — and the arithmetic breaks the other way.

The problem is no longer storing the bytes — it is that per-object overhead stops being negligible. Metadata that was rounding error on a 4 GB video is a large fraction of a 20 KB thumbnail, and the index that has to be searched to find the object no longer fits in memory.

The fix inverts chunking: instead of splitting one blob across many files, pack many blobs into one large file and keep an offset index.

Two things this buys. The file system stops being asked to track billions of files — it tracks thousands of large ones, and the store keeps its own index. And the index entry shrinks to roughly a key, an offset and a length, small enough that billions of them fit in memory, which is what makes a read one seek rather than several.

This is Facebook's Haystack design, and the general shape recurs whenever object count outgrows object size: when per-item overhead dominates the item, stop treating items individually and batch them behind an index. The same instinct as a log-structured store batching small writes into one segment.

Worth volunteering as the boundary condition on everything else in this lesson: chunking is the answer when blobs are too big, packing is the answer when they are too small, and the chunk-size discussion below only applies between those two.

Choosing a chunk size

The system should use a fixed chunk size that balances metadata overhead and storage performance. Smaller chunks increase metadata volume and add load to the manager node. Larger chunks can increase fragmentation and degrade I/O performance. For rotating disks, the chunk size should align with the disk's sector size.

Chunk size too SMALL                    Chunk size too LARGE
--------------------                    --------------------
More chunks per blob                    Fewer chunks per blob
-> more metadata rows                   -> less metadata
-> more load on the manager node        -> more internal fragmentation
-> more parallelism on read             -> less parallelism on read
                                        -> larger unit of re-replication

Put the metadata pressure in numbers — it is the constraint that actually binds

Take Lesson 4's 12.51 TB/day of ingest and price the metadata:

64 MB chunks  : 12.51 TB / 64 MB   ~ 195,000 chunks/day
 1 MB chunks  : 12.51 TB / 1 MB    ~ 12,500,000 chunks/day

Each chunk needs a metadata row with four location IDs. At 1 MB chunks that is 12.5 million new rows every day flowing through a manager node whose ceiling is roughly 10,000 QPS (Lesson 12). At 64 MB it is 195,000 — two orders of magnitude less pressure on the exact component that is already the system's bottleneck.

That is why large object stores use chunks measured in tens of megabytes: the binding constraint is not disk, it is manager-node metadata throughput. GFS chose 64 MB for precisely this reason.

The counter-pressure is real too — very large chunks mean less read parallelism and coarser repair — but at these volumes the metadata term dominates.

'Align with the disk's sector size' is the low-level version of the same idea

On rotating disks, an I/O that straddles sector boundaries costs two physical operations instead of one. Aligning chunk boundaries to sectors means every chunk read is a clean sequential operation.

Same reasoning as pub-sub's append-only segments: match your logical unit to the hardware's physical unit and you stop paying an alignment tax on every operation.

Worth noting it generalizes past spinning disks — on SSDs the equivalent is aligning to erase blocks to avoid write amplification. The mechanism changes; the principle does not.

What about the last chunk?

"What if the blob size isn't a multiple of our configured chunk size? How does the manager node know how many bytes to read for the last chunk?"

If the blob size isn't a multiple of the chunk size, the last chunk won't be full. The manager node also keeps the size of each blob to determine the number of bytes to read for the last chunk.

Small detail, real consequence: because the blob's total size is stored, the manager node can deterministically compute the byte range for every chunk — chunk i covers bytes i × chunkSize to (i+1) × chunkSize − 1, with the last one truncated at the recorded size.

That is what makes range reads possible. A client asking for bytes 100 MB–101 MB can be told exactly which chunk holds them without reading anything, which is the foundation of Lesson 11's streaming.

Interview signal by level

Next: where those chunks and blobs actually live.