Workflows: Write, Read, and Delete

In one line: the read path here does something specific and important — the client reads chunks directly from data nodes, bypassing the frontend entirely. That one decision is what makes Lesson 4's egress numbers survivable.

Writing a blob

1. The client sends an upload request. If it passes the rate limiter, the load balancer forwards it to a frontend server, which asks the manager node where to store the blob.
2. The manager node assigns a unique ID using a unique ID generator system. It splits the blob into fixed-size chunks and assigns them to data nodes using a free-space management system to ensure sufficient storage.
3. The frontend server writes the chunks to the assigned data nodes.
4. The manager node orchestrates chunk replication for redundancy, allocating storage and data nodes for the replicas.
5. The manager node records the blob metadata in the metadata storage.
6. The client receives the fully qualified path — user ID, container ID, blob ID, and access level.

Free-space-aware placement is a different strategy from every other chapter

Notice what decides where a chunk goes: a free-space management system, not consistent hashing.

That is a departure worth explaining. The Distributed Cache and Key-Value Store chapters hashed keys to nodes because placement had to be computable by the client without a lookup. Here, a manager node is consulted on every operation anyway, so placement can be a decision rather than a function.

And a decision is strictly better when nodes are heterogeneous or unevenly full: hashing spreads keys evenly whether or not the targets have room, while free-space-aware placement puts data where the space actually is.

The cost is that placement must be remembered — hence the blob metadata in step 5. Same trade message-queue design drew between computed and directory-based placement: no lookup but constrained distribution, versus arbitrary distribution but a directory to maintain.

Two clients uploading the same blob name concurrently

"What does the manager node do if a user concurrently writes two blobs with the same name inside the same container?"

The manager node serializes these operations and assigns a version number to the uploaded blob.

This is why the manager node's centrality is a feature and not only a liability. Because every write consults it, it is a natural serialization point — there is no distributed agreement to reach, no vector clocks, no last-write-wins ambiguity. One node decides the order.

Compare key-value stores, which needed vector clocks and reconciliation precisely because it had no central coordinator. Centralization costs availability and buys simplicity; the blob store takes that trade and pays for it with checkpointing.

Keeping metadata and bytes in agreement

The write workflow above assumes the manager node sees the upload. Once the bytes go direct to storage on a presigned URL, it does not — and now two systems that update at different times both claim to know whether a blob exists.

The tempting fix is to let the client tell you. It fails four ways, and they are worth naming because interviewers push exactly here:

• Race. The client reports success before the storage service has durably committed, so a reader gets a 404 on a blob the database calls complete.
• Orphans. The client crashes after uploading and before reporting. Bytes exist with no metadata row — invisible, undeletable through the API, and billed indefinitely.
• Malice. A client can claim completion without uploading anything, and now the catalogue lists a blob that does not exist.
• Lost notification. The completion call itself fails on a flaky network, which is exactly the population most likely to have needed multipart in the first place.

The pattern that works is to stop trusting the client and let storage confirm what storage holds:

The object key is the join: the manager node wrote the row with that key before issuing the URL, so the event carries enough to find the exact row. The client is out of the trust path entirely.

Events can still be lost or delayed, which is why the reconciliation sweep is not optional — it is the safety net that turns "usually consistent" into "eventually consistent with a bound". A row that has been pending longer than the URL's lifetime either has bytes behind it, in which case complete it, or does not, in which case expire it.

The honest cost: blob status is eventually consistent now, lagging by an event hop. That is a fair price for not routing petabytes through the application tier, and saying so is better than pretending the direct-upload path is free.

Reading a blob

1. The frontend server receives a read request and requests metadata for the blob from the manager node.
2. The manager node verifies whether the blob is public or private and checks if the user is authorized.
3. If authorized, the manager node returns the chunk mappings (data node locations) to the client.
4. The client reads the chunk data directly from the data nodes.

Note: clients cache metadata to speed up future reads and reduce the load on the manager node.

Step 4 is the most important line in the chapter

The client reads chunks directly from the data nodes. The frontend and manager node are not in the data path.

Lesson 4 computed 462.96 Gb/s of egress. If that flowed through the frontend tier, those servers would carry it twice — once inbound from data nodes, once outbound to clients — for nearly a terabit of traffic through a tier whose job is request handling.

Instead the control plane answers a small question (where are the chunks, may you read them) and steps out of the way. The bulk transfer happens over as many parallel connections as there are chunks, straight from the machines holding them.

Two further wins fall out. Parallelism: a blob in four chunks on four data nodes downloads over four connections at once — Lesson 12's throughput argument. And the manager node's capacity is measured in operations, not bandwidth, which is what keeps its ~10,000 QPS ceiling (Lesson 12) tolerable.

This is the same shape as the pub-sub broker and the rate limiter: the coordinator answers questions; it does not carry payload.

Cached metadata goes stale, and the design handles it by failing

"Suppose the manager node moves data from one data node to another because of an impending disk failure. The user will now have stale information if they use the cached metadata. How do we handle this?"

The client calls fail. The client then flushes the cache and fetches new metadata from the manager node.

This is invalidation by failure, and it is a deliberate, sensible choice rather than a gap.

The alternative — the manager node proactively notifying every client that ever cached a mapping — requires tracking who holds what, and a push channel to every client. That is the same problem distributed caching needed a whole configuration service to solve, and here the client population is unbounded and largely external.

The blob store instead makes the stale case cheap and self-correcting: try, fail, refetch, retry. It works because a stale read fails loudly — the chunk is not there — rather than silently returning wrong data.

Contrast that with distributed caching's rejoining-node hazard, where a stale node returned confident wrong answers. Fail-and-refetch is only safe when staleness is detectable, and here it is.

Deleting a blob

Upon receiving a delete request, the manager node marks the blob as deleted in the metadata. The garbage collector frees the actual storage space asynchronously.

Delete is a metadata write, which is why it's fast

A blob's chunks are spread across many data nodes, each with replicas. Synchronously removing every copy would mean a fan-out of dozens of operations, bounded by the slowest node — Lesson 11 makes this argument in full.

Marking one metadata row is one write, and it satisfies what the user actually asked for: the blob is now inaccessible. Reclaiming the space is an internal concern with no deadline.

The price is that deleted-but-not-reclaimed storage is real, billed capacity, and that metadata is temporarily inconsistent with what data nodes hold. Lesson 11 is about paying that price safely.

Interview signal by level

Next: how blobs are actually split up.