Garbage Collection, Streaming, and Caching
In one line: garbage collection here is a case of deliberately creating an inconsistency to keep a user-facing operation fast, then cleaning it up out of band. That pattern recurs constantly, and this is a clean example of it.
Garbage collection
The workflow, as the design illustrates it:
Disk 1 holds three blobs: [b020, b903, b883] Total storage: 6 GB
Metadata
Disk Info : Disk_ID 1 -> [b020, b903, b883]
Blob Info : b020 -> Disk1
b903 -> Disk1
b883 -> Disk1
1. User requests: delete blob b903
2. Delete Blob API looks up b903 in the Blob Table
3. Entry for b903 is removed so nobody can access it.
The garbage collector picks up the orphaned entry.
4. Acknowledgment returns to the Delete Blob API.
The chunks are still on Disk 1 -- reclaimed later.
Marking a blob as deleted without immediately removing it introduces internal metadata inconsistencies, leaving storage space temporarily occupied. These inconsistencies are invisible to users. Although a blob is marked deleted, its chunk metadata remains, and data nodes continue to store the chunks. A garbage collector later cleans up stale metadata and removes the associated chunks.
The design creates an inconsistency on purpose — and the reason it's safe is that it's one-directional
Deliberately allowing metadata and storage to disagree sounds alarming. It is safe here because the inconsistency only ever runs one way: there is data on disk that nothing points to.
That direction is harmless — orphaned chunks waste space and nothing else. The dangerous direction, metadata pointing at data that no longer exists, never occurs, because the pointer is removed first and the bytes second.
Order matters enormously. Delete the chunks first and you would have a window where a live blob's metadata references missing data — a user-visible corruption. Delete the pointer first and the worst case is a bill for storage nobody is using.
The generalizable rule: when you must be inconsistent, choose the direction whose failure mode is cost rather than correctness.
The consequence is a real, ongoing storage bill
There can be a delay between a blob delete request and the reclamation of the associated storage space. This trade-off reduces delete request latency by deferring physical cleanup.
Two operational consequences worth naming:
Deleted data still costs money. At Lesson 4's ingest rates, a garbage collector running too slowly means paying for terabytes nobody can access. The GC's throughput must exceed the deletion rate or the gap grows without bound.
Deleted data still exists. For a privacy deletion request, "marked deleted" is not "erased" — the bytes are recoverable until the collector runs and, on some media, after. Compliance regimes with hard deletion deadlines need the GC's worst-case latency to be bounded and provable, not best-effort.
Combined with Lesson 3's retention window, the honest timeline is: user deletes → inaccessible immediately → retained for the retention period → collected → space reclaimed. That is several stages between "delete" and "gone," and being able to state them is what a compliance-aware answer sounds like.
Streaming a file
Streaming involves reading a file in sequential parts. If the read size is X bytes: the first request covers bytes 0 to X−1, the second request covers X to 2X−1.
"How do we know which bytes we have read and which to read next?"
We use an offset value to track where to resume reading. The offset starts at 0. If we read X bytes, the next offset becomes X, meaning the next read starts at byte position X and continues to 2X−1. After each read, the offset is updated by adding the number of bytes just read.
offset = 0 read X bytes -> bytes 0 .. X-1 -> offset = X read X bytes -> bytes X .. 2X-1 -> offset = 2X read X bytes -> bytes 2X .. 3X-1 -> offset = 3X
Streaming is what makes a video start playing before it has downloaded
The mechanism is simple; the product consequence is not. Without it, watching a 50 MB video means downloading 50 MB first. With it, the player fetches the first few hundred kilobytes, starts rendering, and continues fetching as it plays.
Two things from earlier lessons make it work. Chunking (Lesson 8) means the file is already split, so a byte range maps to specific chunks. And because the manager node stores the blob's total size, it can compute exactly which chunk holds byte N without reading any data.
Note the symmetry with Lesson 5's multipart upload: read side and write side both operate on ranges rather than whole objects, and both fall out of the same chunked layout. That is what a well-chosen internal structure buys — the API affordances come for free.
Caching
Caching improves performance at multiple levels:
| Layer | What it caches | Why |
|---|---|---|
| Client-side | Chunk metadata | Access data nodes directly for subsequent reads — skips the manager node entirely |
| Frontend servers | Partition maps | Quickly route requests |
| Manager node | Frequently accessed chunks | Reduce disk I/O |
Note: public blobs are often cached using a CDN. The CDN serves content until the TTL defined by the origin server's Cache-Control header expires.
The CDN is the load-bearing cache — the other three are optimizations
Immutability is what makes the edge cache trivial: the bytes at a blob id never change, so there is no invalidation problem to solve.
Lesson 4 computed 462.96 Gb/s of egress. The three internal caches reduce metadata lookups and disk I/O; none of them reduces the bytes leaving the system. Only the CDN does that.
Push popular blobs to the edge and origin egress collapses to the long tail. That is the difference between a viable system and an unaffordable one, and it is why CDNs's building block is essential here rather than optional.
Two properties make blobs unusually CDN-friendly, both from Lesson 1. Blobs are immutable, so a cached copy can never be wrong — there is no invalidation problem, which is normally the hardest part of CDN operation. And they are large and popular in a skewed distribution, so a small hot set serves most of the traffic.
Note the qualifier though: public blobs. Private blobs need authorization on every read, which is what Lesson 6's time-limited signed URLs are for — and they are correspondingly harder to cache at the edge.
Client-side metadata caching is the one that eliminates a whole round trip
The other caches make an existing step faster. Client-side metadata caching removes a step entirely: with mappings cached, the client goes straight to the data nodes without consulting the manager node at all.
That directly relieves the system's known bottleneck — Lesson 12 puts the manager node at roughly 10,000 QPS. Every cached mapping is a request it never sees.
The cost is the staleness handled in Lesson 7: when chunks move, cached mappings break, the call fails, and the client refetches. Invalidation by failure — acceptable precisely because the failure is loud rather than silent.
Key takeaway
Deletion is deferred: remove the pointer first, reclaim bytes later, so the only inconsistency is orphaned data nobody references — cost, not corruption. Offsets make streaming resumable, reusing the same position-not-count idea as pub-sub and pagination. And of four caching layers, only the CDN reduces egress — the metric that actually dominates.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Deletes are asynchronous and we cache and use a CDN." |
| L5 | Explains the deferral: "we mark the blob deleted so it's immediately inaccessible and let a garbage collector reclaim chunks later, because synchronously deleting every replica across data nodes would be slow." |
| Staff+ | Names the ordering and the CDN's role: "the inconsistency is one-directional by design — orphaned chunks nobody points at, never metadata pointing at missing data — so the failure mode is cost, not corruption. The GC's throughput has to exceed the deletion rate or the gap grows, and for privacy deletions its worst-case latency needs to be bounded. On caching, only the CDN reduces egress, which is the metric that dominates — and blobs are unusually CDN-friendly because immutability means there's no invalidation problem." |
Next: checking the design against the requirements.