Evaluation

In one line: the scalability section contains the design's honest limit — the manager node caps a single instance at roughly 10,000 QPS, and the answer is to run more instances rather than to fix it.

Availability

Replication improves availability and fault tolerance. The system maintains four replicas per blob to distribute read traffic and tolerate node failures. The placement strategy spans multiple data centers and regions to reduce the impact of large-scale outages. A monitoring service tracks replica health and triggers re-replication when failures exceed configured thresholds.

For write requests, synchronous replication within the cluster provides strong consistency and improves fault tolerance. To maintain manager node availability, the system persists periodic state snapshots — if the active node fails, a standby instance can be brought up from the persisted state.

Four replicas here, three replicas in Lesson 8 — same thing, different convention

Lesson 8's metadata table shows Datanode ID plus three replica IDs, and says "three replicas per chunk." This lesson says "four replicas per blob."

Both are correct: three replicas in addition to the primary is four copies total. The evaluation counts the primary; the metadata table names it separately.

Keep it distinct from Lesson 10's three geographic tiers — local, second data center, different region — which is a placement policy across failure domains, not a copy count.

The phrasing that survives any follow-up: four copies of every chunk, placed across at least three failure domains.

'Triggers re-replication when failures exceed thresholds' is the self-healing loop

This is the part that makes durability sustainable rather than a one-time property.

Replicas decay — disks die, nodes are decommissioned, racks lose power. Without automated repair, replica count only ever goes down, and the system drifts toward data loss silently. The monitoring service closing that loop is what keeps the copy count at its target indefinitely.

Note the interaction with Lesson 10: because replication is per chunk, re-replication sources from many nodes in parallel, so the under-replicated window is short. That window is exactly when a second failure loses data, so shortening it is the single highest-leverage durability lever — more so than adding a fourth copy.

Durability

Synchronous replication within the storage cluster guarantees data durability. If a node loses data, it is recovered from peers. The monitoring service tracks disk health; upon failure it alerts administrators and triggers the manager node to copy affected content to healthy disks, updating mappings accordingly.

Replication only helps if you know which copy is right

The paragraph above handles a disk that fails — it stops responding, monitoring notices, the manager re-replicates. That is the easy failure, because it announces itself.

The hard one is a disk that keeps working and quietly returns the wrong bytes. Bit rot, a firmware bug, a bad cable, a cosmic ray: the read succeeds, the data is corrupt, and nothing anywhere reports an error.

Three replicas are no defence on their own. With three differing copies and no way to tell which is correct, you have three candidates and no answer — and if repair copies the corrupt one over the healthy ones, replication actively spreads the damage.

The fix is that every chunk carries a checksum written at upload time, and it is used at three moments:

The background scrub is the part candidates miss, and it is the one that makes archival storage credible. A blob nobody reads for three years is exactly where silent corruption would go undetected until the moment it is finally needed. Scrubbing turns "we hope it is still there" into "we verified it last week", and a failed check is repaired from a replica or reconstructed from erasure-coded fragments before it becomes a second failure.

This is also where the eleven nines actually come from. 99.999999999% is not a claim about disks — consumer drives are nowhere near that. It is what you get by combining independent copies across failure domains with continuous verification, so that losing an object requires several independent failures inside the window before repair completes.

The general statement worth carrying: redundancy protects against loss you can detect; checksums are what make the loss detectable. Without them, replication is a way of storing three unverified opinions.

Scalability

Splitting blobs into chunks and partitioning them by range allows the system to scale to billions of requests. Partition servers handle specific ranges, providing automatic load balancing. Storage scales horizontally by adding data nodes.

However, the centralized manager node creates a potential bottleneck, with a capacity of approximately 10,000 QPS.

"How can we further scale when our manager server reaches its limits and we can't improve its performance through vertical scaling?"

We can make two independent instances of our system. Each instance will have its own manager node and a set of data nodes. Deployment of a system similar to ours has been shown to scale up to a few petabytes, so making additional instances can help us scale further.

For further scaling inside a single instance, we need a new, more complicated design.

10,000 QPS is small — and that number is the whole reason the design looks like this

Compare with Lesson 4's estimate of 5 million requests per second at peak. The manager node handles 10,000. That is a factor of 500 short.

The design survives only because most requests never touch the manager node:

• Client-side metadata caching (Lesson 11) lets repeat reads go straight to data nodes.
• Direct client-to-data-node reads (Lesson 7) keep bulk traffic entirely off the control path.
• Large chunks (Lesson 8) minimize metadata operations per byte stored.
• The CDN (Lesson 11) absorbs popular reads before they reach the system at all.

Every one of those is an optimization around the manager node's ceiling. Seen that way, the architecture is largely organized to protect one component from the traffic — which is a genuinely useful way to describe it in an interview.

'Make two independent instances' is federation, and it has real costs

The answer to outgrowing the manager node is horizontal scaling of whole systems, not of the manager node. That works, and it is what the design recommends — but be honest about what it costs:

• No cross-instance operations. An account on instance A cannot have a container on instance B, and there is no global listing.
• Something must route. A directory mapping accounts to instances is now required — and that directory becomes the new global component to keep available.
• Rebalancing is expensive. Moving a large account between instances means moving its petabytes.

The source's closing line is the important admission: "for further scaling inside a single instance, we need a new, more complicated design." That is where systems like Facebook's Tectonic go — disaggregating metadata so it scales independently rather than living on one node.

Recognizing that federation is a workaround for a centralization limit, not a solution to it, is the senior read.

Throughput

Distributing chunks across multiple data nodes enables parallel fetching, significantly increasing throughput. Caching at the client, front-end, and manager node layers further improves performance and reduces latency.

Reliability

Heartbeat protocols allow the manager node to monitor data node health and route requests only to healthy instances. When a node fails, the system triggers re-replication to restore the desired replica count. Monitoring services alert operators to hardware failures — disk or switch issues — and to low disk space conditions.

Consistency

We achieve strong consistency within the storage cluster by synchronously replicating data blocks during the write request (critical path). Subsequent reads are served from this cluster until the data is asynchronously replicated to remote data centers for regional availability.

Read that carefully — consistency is scoped to the cluster, and reads are pinned to keep it

Two mechanisms working together:

Synchronous replication within the cluster means a write is not acknowledged until the local copies exist, so any subsequent read of those copies sees it.

Reads served from the primary cluster is the part that makes the guarantee hold. Cross-region copies lag, so serving a read from a remote region could return stale data. Pinning reads to the cluster that has the synchronous copies preserves strong consistency — at the cost of remote-region reads being slower or unavailable until replication catches up.

This is the same resolution distributed caching reached: strong consistency within a failure domain, eventual across them, and route reads to preserve the stronger guarantee where it matters.

What the design gives up — worth volunteering

• No cross-region strong consistency. Remote copies lag, so a regional disaster can lose blobs written in the replication window.
• A single instance is bounded by manager-node QPS and by "a few petabytes"; beyond that it is federation with no cross-instance operations.
• Deleted data persists until the garbage collector runs — a cost and a compliance consideration.
• Range partitioning risks hotspots on large or busy accounts, which hashing would have avoided.
• No in-place updates. Immutability is a hard constraint, not a default.
• Blob status is eventually consistent once uploads go direct to storage — it lags an event hop, with a reconciliation sweep as the bound.
• Erasure coding is not free. Where it replaces replication it buys disk back at the cost of read and repair amplification, so it belongs on cold data only.

Interview signal by level

Next: the whole design under interview conditions.