Temporary Failures: Sloppy Quorums and Hinted Handoff
Why this matters: a strict quorum turns node failure into write failure, which directly contradicts the always-write requirement. This is the mechanism that resolves that contradiction — and it is where "highly available" stops being a slogan.
Key takeaway
In a sloppy quorum, the first n healthy nodes from the preference list handle reads and writes. These may not be the designated owners in the consistent hash ring, but they ensure the request is processed.
The problem with a strict quorum
Many distributed systems use a strict read/write quorum: an operation must receive responses from a minimum number of replicas before it can proceed. If enough replicas are unavailable and the quorum cannot be satisfied, the operation fails — reducing availability.
That is correct behavior for a CP system. It is unacceptable here, because Lesson 2 made "always write" a functional requirement.
Sloppy quorums
The relaxation is simple: rather than requiring the designated n nodes, take the first n healthy nodes walking the preference list.
| Strict quorum | Sloppy quorum | |
|---|---|---|
| Which nodes count | The designated owners only | The first n healthy nodes in the preference list |
| If owners are down | Operation fails | Operation succeeds on substitutes |
| Availability | Lower | Higher |
| Consistency guarantee | r + w > n overlap holds | Overlap may not hold during the failure |
That last row is the honest cost, and worth stating plainly: during a failure the read and write sets can land on different nodes, so a read may miss a recent write until the data finds its way home.
Hinted handoff
The mechanism that gets it home:
Example. Consider n = 3. If Node A is unavailable during a write, the request is sent to the next healthy node, Node D. Node D processes the request and stores a hint indicating the data belongs to A. Once A recovers, D forwards the data to A and removes the local copy.
This approach is called hinted handoff. It ensures that reads and writes are fulfilled even if a node faces a temporary failure.
Key takeaway
A sloppy quorum trades the r + w > n guarantee during a failure for the ability to keep accepting requests. Hinted handoff is what makes that trade temporary rather than permanent — the write is parked, not abandoned.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "If a node is down we write to another one." |
| L5 | Names both parts: "sloppy quorum takes the first n healthy nodes, and hinted handoff sends the data back to the real owner when it recovers." |
| Staff+ | States the cost and the boundary: "during the failure the read and write sets may not overlap, so r + w > n doesn't hold — that's the price of staying writable. And this only covers temporary failures; a node that never returns needs anti-entropy instead." |
Next: repairing replicas after a permanent failure.