Temporary Failures: Sloppy Quorums and Hinted Handoff
In one line: a strict quorum turns node failure into write failure, which directly contradicts the always-write requirement. This is the mechanism that resolves that contradiction — and it is where "highly available" stops being a slogan.
The problem with a strict quorum
Many distributed systems use a strict read/write quorum: an operation must receive responses from a minimum number of replicas before it can proceed. If enough replicas are unavailable and the quorum cannot be satisfied, the operation fails — reducing availability.
That is correct behavior for a CP system. It is unacceptable here, because Lesson 2 made "always write" a functional requirement.
Sloppy quorums
The relaxation is simple: rather than requiring the designated n nodes, take the first n healthy nodes walking the preference list.
| Strict quorum | Sloppy quorum | |
|---|---|---|
| Which nodes count | The designated owners only | The first n healthy nodes in the preference list |
| If owners are down | Operation fails | Operation succeeds on substitutes |
| Availability | Lower | Higher |
| Consistency guarantee | r + w > n overlap holds | Overlap may not hold during the failure |
That last row is the honest cost, and worth stating plainly: during a failure the read and write sets can land on different nodes, so a read may miss a recent write until the data finds its way home.
Hinted handoff
The mechanism that gets it home:
Example. Consider n = 3. If Node A is unavailable during a write, the request is sent to the next healthy node, Node D. Node D processes the request and stores a hint indicating the data belongs to A. Once A recovers, D forwards the data to A and removes the local copy.
This approach is called hinted handoff. It ensures that reads and writes are fulfilled even if a node faces a temporary failure.
Why the hint matters
Without the hint, D would simply hold data it does not own. The ring says that key belongs to A, so once A returns, reads route to A — which never received the write. The write would be silently lost despite having been acknowledged.
The hint is what makes the handoff temporary and self-correcting: it records the debt and pays it back automatically. The data is not just stored somewhere convenient; it is stored with a note saying where it should be.
What hinted handoff does not cover
It handles temporary failures — a node that comes back. If the node never returns, or returns after so long that its data is far out of date, the hint has nowhere useful to go and the replicas remain divergent.
That is a permanent failure, and it needs a different mechanism: anti-entropy with Merkle trees, in the next lesson. Distinguishing the two cases — and having a separate answer for each — is exactly the structure an interviewer is listening for.
Key takeaway
A sloppy quorum trades the r + w > n guarantee during a failure for the ability to keep accepting requests. Hinted handoff is what makes that trade temporary rather than permanent — the write is parked, not abandoned.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "If a node is down we write to another one." |
| L5 | Names both parts: "sloppy quorum takes the first n healthy nodes, and hinted handoff sends the data back to the real owner when it recovers." |
| Staff+ | States the cost and the boundary: "during the failure the read and write sets may not overlap, so r + w > n doesn't hold — that's the price of staying writable. And this only covers temporary failures; a node that never returns needs anti-entropy instead." |
Next: repairing replicas after a permanent failure.